[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:21:21 dist-test-slave-5kzz krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:21:21 dist-test-slave-5kzz krb5kdc[395](info): set up 2 sockets
Jun 20 23:21:21 dist-test-slave-5kzz krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 23:21:22.951264 388 mini_kdc.cc:121] Time spent starting KDC: real 2.326s user 0.006s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:21:23 dist-test-slave-5kzz krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:21:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:21:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:21:23.112638 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36217
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:23.430485 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:23.430888 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:23.431041 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:23.438871 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:21:23.439011 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:23.439111 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:23.439216 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:21:23.439289 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:21:23.448975 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:23.450250 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:23.451638 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:23.463860 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:23.463891 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:23.464197 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:23.464934 408 server_base.cc:1034] running on GCE node
I20260620 23:21:23.465675 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:23.467168 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:23.468384 408 hybrid_clock.cc:648] HybridClock initialized: now 1781997683468295 us; error 84 us; skew 500 ppm
I20260620 23:21:23.468730 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:23 dist-test-slave-5kzz krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:23.471804 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:21:23.473250 408 webserver.cc:460] Webserver started at http://127.0.97.62:45751/ using document root <none> and password file <none>
I20260620 23:21:23.473933 408 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:23.474072 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:23.474367 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:23.477283 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "a95dee564170441faead2934939cd875"
format_stamp: "Formatted at 2026-06-20 23:21:23 on dist-test-slave-5kzz"
server_key: "8487e41b0138aa2ae67ee761874f2e72"
server_key_iv: "df3bc1487ce060eeabb1fe776d660b9e"
server_key_version: "encryptionkey@0"
I20260620 23:21:23.478415 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "a95dee564170441faead2934939cd875"
format_stamp: "Formatted at 2026-06-20 23:21:23 on dist-test-slave-5kzz"
server_key: "8487e41b0138aa2ae67ee761874f2e72"
server_key_iv: "df3bc1487ce060eeabb1fe776d660b9e"
server_key_version: "encryptionkey@0"
I20260620 23:21:23.483512 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 23:21:23.486665 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:23.487939 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:21:23.488123 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "a95dee564170441faead2934939cd875"
format_stamp: "Formatted at 2026-06-20 23:21:23 on dist-test-slave-5kzz"
server_key: "8487e41b0138aa2ae67ee761874f2e72"
server_key_iv: "df3bc1487ce060eeabb1fe776d660b9e"
server_key_version: "encryptionkey@0"
I20260620 23:21:23.488310 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:23.509454 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:23.514302 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:23.514619 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:23.533363 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36217
I20260620 23:21:23.533402 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36217 every 8 connection(s)
I20260620 23:21:23.534785 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:21:23.537655 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 408
I20260620 23:21:23.537943 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:21:23.538967 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:23.539892 388 external_mini_cluster.cc:1428] Setting key aeadce312b128000cc54cd4bad650458
I20260620 23:21:23.550036 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875: Bootstrap starting.
I20260620 23:21:23.553022 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:23.554101 475 log.cc:824] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:23.556874 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875: No bootstrap required, opened a new log
I20260620 23:21:23.564184 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95dee564170441faead2934939cd875" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36217 } }
I20260620 23:21:23.564560 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:23.564672 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95dee564170441faead2934939cd875, State: Initialized, Role: FOLLOWER
I20260620 23:21:23.565326 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95dee564170441faead2934939cd875" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36217 } }
I20260620 23:21:23.565598 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:21:23.565732 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:21:23.565924 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 23:21:23 dist-test-slave-5kzz krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:23.569037 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95dee564170441faead2934939cd875" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36217 } }
I20260620 23:21:23.569657 475 leader_election.cc:302] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a95dee564170441faead2934939cd875; no voters:
I20260620 23:21:23.574826 475 leader_election.cc:288] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:21:23.575093 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:23.581019 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [term 1 LEADER]: Becoming Leader. State: Replica: a95dee564170441faead2934939cd875, State: Running, Role: LEADER
I20260620 23:21:23.581732 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95dee564170441faead2934939cd875" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36217 } }
I20260620 23:21:23.581980 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:21:23.585933 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:23.543709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.0.97.62:36217)
0620 23:21:23.545664 (+ 1955us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:23.545678 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:23.545728 (+ 50us) server_negotiation.cc:407] Connection header received
0620 23:21:23.554971 (+ 9243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:23.555006 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:23.555508 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:23.555892 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:23.558759 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:23.560529 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:23.562130 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:23.562879 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:23.563097 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:23.569780 (+ 6683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:23.569788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:23.569835 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:23.569887 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:23.579833 (+ 9946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:23.581220 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:23.581229 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:23.581237 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:23.581462 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:23.582072 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:23.582076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:23.582079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:23.582801 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:23.583081 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:23.584324 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0620 23:21:23.584749 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1740,"thread_start_us":1513,"threads_started":1}
I20260620 23:21:23.587553 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a95dee564170441faead2934939cd875. Latest consensus state: current_term: 1 leader_uuid: "a95dee564170441faead2934939cd875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95dee564170441faead2934939cd875" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36217 } } }
I20260620 23:21:23.587673 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a95dee564170441faead2934939cd875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95dee564170441faead2934939cd875" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36217 } } }
I20260620 23:21:23.588006 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:23.588032 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875 [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:23.590610 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:21:23.596798 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:21:23.605710 487 catalog_manager.cc:1347] Generated new cluster ID: b6fefb7c1243401cad453044379fb543
I20260620 23:21:23.605845 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:21:23.616230 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:21:23.617498 487 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:21:23.623095 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a95dee564170441faead2934939cd875: Generated new TSK 0
I20260620 23:21:23.623874 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:21:23.666349 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36217
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:23.998258 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:23.998539 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:23.998726 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:24.006026 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:24.006170 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:24.006371 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:21:24.016685 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36217
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:24.017995 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:24.019412 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:24.032744 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:24.032907 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:24.032941 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:24.033846 501 server_base.cc:1034] running on GCE node
I20260620 23:21:24.034464 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:24.035439 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:24.036659 501 hybrid_clock.cc:648] HybridClock initialized: now 1781997684036613 us; error 39 us; skew 500 ppm
I20260620 23:21:24.036970 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:24 dist-test-slave-5kzz krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:24.040108 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:21:24.041729 501 webserver.cc:460] Webserver started at http://127.0.97.1:38305/ using document root <none> and password file <none>
I20260620 23:21:24.042362 501 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:24.042454 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:24.042757 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:24.045521 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "f0576ba94e894c1da3775ed7db01dccd"
format_stamp: "Formatted at 2026-06-20 23:21:24 on dist-test-slave-5kzz"
server_key: "f04e80cf5b04da7d65cbb9ad961a6570"
server_key_iv: "74b9a330bd28c87f709e36a63b13472d"
server_key_version: "encryptionkey@0"
I20260620 23:21:24.046227 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f0576ba94e894c1da3775ed7db01dccd"
format_stamp: "Formatted at 2026-06-20 23:21:24 on dist-test-slave-5kzz"
server_key: "f04e80cf5b04da7d65cbb9ad961a6570"
server_key_iv: "74b9a330bd28c87f709e36a63b13472d"
server_key_version: "encryptionkey@0"
I20260620 23:21:24.051198 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 23:21:24.055051 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:24.056432 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260620 23:21:24.056600 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "f0576ba94e894c1da3775ed7db01dccd"
format_stamp: "Formatted at 2026-06-20 23:21:24 on dist-test-slave-5kzz"
server_key: "f04e80cf5b04da7d65cbb9ad961a6570"
server_key_iv: "74b9a330bd28c87f709e36a63b13472d"
server_key_version: "encryptionkey@0"
I20260620 23:21:24.056793 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:24.071825 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:24.076292 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:24.076587 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:24.077546 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:24.078892 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:24.078962 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:24.079244 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:24.079330 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:24.115468 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41617
I20260620 23:21:24.115499 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41617 every 8 connection(s)
I20260620 23:21:24.116765 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:21:24.123302 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 501
I20260620 23:21:24.123533 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:21:24.124106 388 external_mini_cluster.cc:1428] Setting key da64aae5712ef0574fe19387bc304f5a
Jun 20 23:21:24 dist-test-slave-5kzz krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:24.137404 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:24.119371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48061 (local address 127.0.97.62:36217)
0620 23:21:24.119794 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:24.119798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:24.120606 (+ 808us) server_negotiation.cc:407] Connection header received
0620 23:21:24.122490 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:24.122496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:24.122582 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:24.122740 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:24.124436 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:24.125329 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:24.127066 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:24.127461 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:24.127598 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:24.131226 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:24.131231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:24.131235 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:24.131265 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:24.133732 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:24.134669 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:24.134673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:24.134726 (+ 53us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:24.134839 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:24.135344 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:24.135348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:24.135352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:24.135645 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:24.135753 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:24.136995 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0620 23:21:24.137166 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":233,"threads_started":1}
I20260620 23:21:24.139065 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:24.119781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36217 (local address 127.0.97.1:48061)
0620 23:21:24.120432 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:24.120487 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:24.122161 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:24.122982 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:24.122989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:24.123695 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:24.124281 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:24.124299 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:24.125504 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:24.125509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:24.126931 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:24.126942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:24.127685 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:24.127692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:24.127931 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:24.128616 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:24.128639 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:24.131011 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:24.133869 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:24.133880 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:24.133900 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:24.134558 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:24.134948 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:24.134955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:24.134962 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:24.135204 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:24.136093 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:24.136103 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:24.136575 (+ 472us) client_negotiation.cc:769] Sending connection context
0620 23:21:24.136925 (+ 350us) client_negotiation.cc:240] Negotiation successful
0620 23:21:24.137558 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":256,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:21:24.140745 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36217
I20260620 23:21:24.141104 629 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:24.141916 629 heartbeater.cc:511] Master 127.0.97.62:36217 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:21:24.144071 440 ts_manager.cc:194] Registered new tserver with Master: f0576ba94e894c1da3775ed7db01dccd (127.0.97.1:41617)
I20260620 23:21:24.145490 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:21:24.161980 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36217
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:24.481037 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:24.481326 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:24.481501 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:24.488629 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:24.488763 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:24.488976 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:21:24.499136 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36217
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:24.500401 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:24.501698 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:24.514287 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:24.514566 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:24.514596 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:24.515954 635 server_base.cc:1034] running on GCE node
I20260620 23:21:24.516623 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:24.517567 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:24.518805 635 hybrid_clock.cc:648] HybridClock initialized: now 1781997684518736 us; error 55 us; skew 500 ppm
I20260620 23:21:24.519080 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:24 dist-test-slave-5kzz krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:24.522102 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:21:24.523481 635 webserver.cc:460] Webserver started at http://127.0.97.2:36301/ using document root <none> and password file <none>
I20260620 23:21:24.524114 635 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:24.524204 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:24.524482 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:24.527387 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "82673a192ff44a3788254dc68f223d7b"
format_stamp: "Formatted at 2026-06-20 23:21:24 on dist-test-slave-5kzz"
server_key: "bad16eb7a9c0070a767fa6a2157b3516"
server_key_iv: "ce730e825f6796a17c25d64e2c9281ba"
server_key_version: "encryptionkey@0"
I20260620 23:21:24.528143 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "82673a192ff44a3788254dc68f223d7b"
format_stamp: "Formatted at 2026-06-20 23:21:24 on dist-test-slave-5kzz"
server_key: "bad16eb7a9c0070a767fa6a2157b3516"
server_key_iv: "ce730e825f6796a17c25d64e2c9281ba"
server_key_version: "encryptionkey@0"
I20260620 23:21:24.533185 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:21:24.536979 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:24.538364 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:21:24.538519 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "82673a192ff44a3788254dc68f223d7b"
format_stamp: "Formatted at 2026-06-20 23:21:24 on dist-test-slave-5kzz"
server_key: "bad16eb7a9c0070a767fa6a2157b3516"
server_key_iv: "ce730e825f6796a17c25d64e2c9281ba"
server_key_version: "encryptionkey@0"
I20260620 23:21:24.538734 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:24.556793 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:24.561004 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:24.561311 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:24.562206 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:24.563560 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:24.563661 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:24.563814 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:24.563890 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:24.609634 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41693
I20260620 23:21:24.610095 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41693 every 8 connection(s)
I20260620 23:21:24.611231 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:21:24.617408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 635
I20260620 23:21:24.617623 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:21:24.618139 388 external_mini_cluster.cc:1428] Setting key 90fb449d83ea2d205c558c883f511f3c
Jun 20 23:21:24 dist-test-slave-5kzz krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:24.632108 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:24.613780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51553 (local address 127.0.97.62:36217)
0620 23:21:24.613967 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:24.613972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:24.615117 (+ 1145us) server_negotiation.cc:407] Connection header received
0620 23:21:24.616828 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:24.616832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:24.616895 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:24.616978 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:24.618981 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:24.619720 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:24.621259 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:24.621662 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:24.621792 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:24.626211 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:24.626216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:24.626219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:24.626250 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:24.628768 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:24.629668 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:24.629672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:24.629674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:24.629801 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:24.630333 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:24.630337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:24.630339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:24.630600 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:24.630771 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:24.631697 (+ 926us) server_negotiation.cc:299] Negotiation successful
0620 23:21:24.631873 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:21:24.633764 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:24.614244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36217 (local address 127.0.97.2:51553)
0620 23:21:24.614964 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:24.615008 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:24.616594 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:24.617256 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:24.617267 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:24.618002 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:24.618814 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:24.618835 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:24.619881 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:24.619887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:24.621111 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:24.621122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:24.621896 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:24.621903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:24.622129 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:24.622837 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:24.622860 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:24.625985 (+ 3125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:24.628914 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:24.628926 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:24.628946 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:24.629552 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:24.629930 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:24.629937 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:24.629944 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:24.630204 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:24.630765 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:24.630776 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:24.631270 (+ 494us) client_negotiation.cc:769] Sending connection context
0620 23:21:24.631631 (+ 361us) client_negotiation.cc:240] Negotiation successful
0620 23:21:24.632251 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":247,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:21:24.635241 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36217
I20260620 23:21:24.635557 763 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:24.636287 763 heartbeater.cc:511] Master 127.0.97.62:36217 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 23:21:24.637853 440 ts_manager.cc:194] Registered new tserver with Master: 82673a192ff44a3788254dc68f223d7b (127.0.97.2:41693)
I20260620 23:21:24.638648 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:21:24.655679 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36217
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:24.978919 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:24.979209 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:24.979393 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:24.987177 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:24.987298 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:24.987533 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:21:24.998128 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36217
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:24.999483 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:25.000802 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:25.014328 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:25.014251 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:25.014302 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:25.015054 768 server_base.cc:1034] running on GCE node
I20260620 23:21:25.015635 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:25.016526 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:25.017750 768 hybrid_clock.cc:648] HybridClock initialized: now 1781997685017709 us; error 53 us; skew 500 ppm
I20260620 23:21:25.018038 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:25 dist-test-slave-5kzz krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:25.021534 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:21:25.023042 768 webserver.cc:460] Webserver started at http://127.0.97.3:46479/ using document root <none> and password file <none>
I20260620 23:21:25.023655 768 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:25.023746 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:25.023977 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:25.026783 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "9144f4686dfd48b29bc28d57e14856f0"
format_stamp: "Formatted at 2026-06-20 23:21:25 on dist-test-slave-5kzz"
server_key: "d616c0089ad3ecc2cd1e82d58a21ed39"
server_key_iv: "0ca19f8e0551041e9db7a8fdc51c488e"
server_key_version: "encryptionkey@0"
I20260620 23:21:25.027635 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9144f4686dfd48b29bc28d57e14856f0"
format_stamp: "Formatted at 2026-06-20 23:21:25 on dist-test-slave-5kzz"
server_key: "d616c0089ad3ecc2cd1e82d58a21ed39"
server_key_iv: "0ca19f8e0551041e9db7a8fdc51c488e"
server_key_version: "encryptionkey@0"
I20260620 23:21:25.032624 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 23:21:25.035950 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:25.037369 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:21:25.037496 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "9144f4686dfd48b29bc28d57e14856f0"
format_stamp: "Formatted at 2026-06-20 23:21:25 on dist-test-slave-5kzz"
server_key: "d616c0089ad3ecc2cd1e82d58a21ed39"
server_key_iv: "0ca19f8e0551041e9db7a8fdc51c488e"
server_key_version: "encryptionkey@0"
I20260620 23:21:25.037675 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:25.058926 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:25.063912 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:25.064256 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:25.065279 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:25.066761 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:25.066865 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:25.067042 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:25.067128 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:25.105592 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45693
I20260620 23:21:25.105636 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45693 every 8 connection(s)
I20260620 23:21:25.106920 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:21:25.112336 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 768
I20260620 23:21:25.112535 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:21:25.112949 388 external_mini_cluster.cc:1428] Setting key fc3cea22b0f9c6e8e734a8ffa00bc713
Jun 20 23:21:25 dist-test-slave-5kzz krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:25.126647 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:25.109495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39771 (local address 127.0.97.62:36217)
0620 23:21:25.109682 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:25.109686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:25.110881 (+ 1195us) server_negotiation.cc:407] Connection header received
0620 23:21:25.113295 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:25.113301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:25.113391 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:25.113525 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:25.115253 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.116098 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.117802 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.118447 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.118573 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:25.121809 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:25.121814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:25.121817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:25.121849 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:25.124239 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:25.124851 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:25.124854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:25.124857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:25.124965 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:25.125310 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:25.125313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:25.125315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:25.125557 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:25.125663 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:25.126275 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 23:21:25.126423 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:21:25.127688 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:25.109992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36217 (local address 127.0.97.3:39771)
0620 23:21:25.110654 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:25.110748 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:25.112963 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:25.113735 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:25.113743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:25.114265 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:25.115069 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:25.115097 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.116318 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.116322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:25.117648 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:25.117659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.118665 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.118672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:25.118901 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:25.119585 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:25.119608 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:25.121618 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:25.124359 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:25.124365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:25.124376 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:25.124749 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:25.125049 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:25.125052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:25.125054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:25.125205 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:25.125655 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:25.125661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:25.125930 (+ 269us) client_negotiation.cc:769] Sending connection context
0620 23:21:25.126163 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 23:21:25.126511 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":259,"threads_started":1}
I20260620 23:21:25.129171 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36217
I20260620 23:21:25.129427 896 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:25.130031 896 heartbeater.cc:511] Master 127.0.97.62:36217 requested a full tablet report, sending...
I20260620 23:21:25.131354 439 ts_manager.cc:194] Registered new tserver with Master: 9144f4686dfd48b29bc28d57e14856f0 (127.0.97.3:45693)
I20260620 23:21:25.132122 439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39771
I20260620 23:21:25.140924 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:21:25.148911 629 heartbeater.cc:503] Master 127.0.97.62:36217 was elected leader, sending a full tablet report...
Jun 20 23:21:25 dist-test-slave-5kzz krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997685, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:21:25 dist-test-slave-5kzz krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997685, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:21:25.183465 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:25.168976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.0.97.1:41617)
0620 23:21:25.169584 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:25.169591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:25.169620 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:21:25.169755 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:25.169767 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:25.169990 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:25.170117 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:25.171486 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.172479 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.173691 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.174242 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.174340 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:25.177964 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:25.177974 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:25.177988 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:25.178019 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:25.180981 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:25.181545 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:25.181551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:25.181553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:25.181658 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:25.181979 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:25.181982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:25.181984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:25.182361 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:25.182566 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:25.182849 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 23:21:25.183033 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":297,"threads_started":1}
W20260620 23:21:25.184636 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45214
Jun 20 23:21:25 dist-test-slave-5kzz krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:21:25 dist-test-slave-5kzz krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:21:25.226836 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:25.213000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45218 (local address 127.0.97.1:41617)
0620 23:21:25.213232 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:25.213238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:25.213510 (+ 272us) server_negotiation.cc:407] Connection header received
0620 23:21:25.213730 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:25.213734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:25.213795 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:25.213910 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:25.215266 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.215931 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.217162 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.217561 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.217684 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:25.221396 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:25.221401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:25.221405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:25.221436 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:25.224473 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:25.225063 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:25.225068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:25.225071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:25.225188 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:25.225530 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:25.225536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:25.225539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:25.225771 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:25.226004 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:25.226299 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 23:21:25.226504 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:21:25.228363 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 501
I20260620 23:21:25.240172 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:25.229314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45222 (local address 127.0.97.1:41617)
0620 23:21:25.229451 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:25.229455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:25.229858 (+ 403us) server_negotiation.cc:407] Connection header received
0620 23:21:25.229954 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:25.229957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:25.230006 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:25.230080 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:25.231206 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.231867 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.233077 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.233486 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.233576 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:25.235665 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:25.235670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:25.235673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:25.235700 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:25.238128 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:25.238649 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:25.238653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:25.238655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:25.238832 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:25.239124 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:25.239127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:25.239129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:25.239345 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:25.239507 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:25.239708 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 23:21:25.239854 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:21:25.241089 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:25.323776 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 501
I20260620 23:21:25.339735 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 635
Jun 20 23:21:25 dist-test-slave-5kzz krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:21:25.355033 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:25.340708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.0.97.2:41693)
0620 23:21:25.341333 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:25.341344 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:25.341381 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:21:25.341442 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:25.341448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:25.341667 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:25.341797 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:25.343049 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.344224 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.345410 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.345941 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.346034 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:25.350133 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:25.350144 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:25.350158 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:25.350189 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:25.352755 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:25.353328 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:25.353334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:25.353337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:25.353446 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:25.353776 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:25.353779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:25.353780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:25.354083 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:25.354317 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:25.354511 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 23:21:25.354750 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":334,"threads_started":1}
I20260620 23:21:25.356244 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:25.437947 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 635
I20260620 23:21:25.460476 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 768
Jun 20 23:21:25 dist-test-slave-5kzz krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:21:25.475857 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:25.461504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43708 (local address 127.0.97.3:45693)
0620 23:21:25.462213 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:25.462220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:25.462249 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:21:25.462313 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:25.462319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:25.462536 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:25.462667 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:25.463881 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.464885 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.466074 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:25.466805 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:25.466901 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:25.470976 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:25.470986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:25.471000 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:25.471033 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:25.473521 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:25.474091 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:25.474098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:25.474100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:25.474208 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:25.474501 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:25.474504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:25.474506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:25.474933 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:25.475150 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:25.475389 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 23:21:25.475573 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":408,"threads_started":1}
I20260620 23:21:25.477100 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:25.558557 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 768
I20260620 23:21:25.577109 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 408
I20260620 23:21:25.578473 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:25.689711 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 408
2026-06-20T23:21:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5117 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:21:25 dist-test-slave-5kzz krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:21:25 dist-test-slave-5kzz krb5kdc[917](info): set up 2 sockets
Jun 20 23:21:25 dist-test-slave-5kzz krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 23:21:26.762980 388 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:21:26 dist-test-slave-5kzz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:21:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:21:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:21:26.927728 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38875
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:27.245265 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:27.245570 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:27.245747 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:27.252962 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:21:27.253065 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:27.253172 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:27.253278 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:21:27.253357 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:21:27.263053 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:27.264387 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:27.265681 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:27.277315 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:27.277458 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:27.277292 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:27.278030 930 server_base.cc:1034] running on GCE node
I20260620 23:21:27.278919 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:27.280265 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:27.281462 930 hybrid_clock.cc:648] HybridClock initialized: now 1781997687281427 us; error 31 us; skew 500 ppm
I20260620 23:21:27.281849 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:27 dist-test-slave-5kzz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:27.288023 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:21:27.289382 930 webserver.cc:460] Webserver started at http://127.0.97.62:38833/ using document root <none> and password file <none>
I20260620 23:21:27.290010 930 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:27.290102 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:27.290343 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:27.293202 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "cbe9796d9eb54195b7ff06d61ef181bf"
format_stamp: "Formatted at 2026-06-20 23:21:27 on dist-test-slave-5kzz"
server_key: "63b51660862ff5457c757827b65670dd"
server_key_iv: "a455082d9cd2d51ee37aa66855f19bd4"
server_key_version: "encryptionkey@0"
I20260620 23:21:27.293887 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "cbe9796d9eb54195b7ff06d61ef181bf"
format_stamp: "Formatted at 2026-06-20 23:21:27 on dist-test-slave-5kzz"
server_key: "63b51660862ff5457c757827b65670dd"
server_key_iv: "a455082d9cd2d51ee37aa66855f19bd4"
server_key_version: "encryptionkey@0"
I20260620 23:21:27.298740 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 23:21:27.301950 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:27.303246 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:21:27.303426 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "cbe9796d9eb54195b7ff06d61ef181bf"
format_stamp: "Formatted at 2026-06-20 23:21:27 on dist-test-slave-5kzz"
server_key: "63b51660862ff5457c757827b65670dd"
server_key_iv: "a455082d9cd2d51ee37aa66855f19bd4"
server_key_version: "encryptionkey@0"
I20260620 23:21:27.303617 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:27.314529 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:27.318231 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:27.318532 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:27.336448 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38875
I20260620 23:21:27.336476 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38875 every 8 connection(s)
I20260620 23:21:27.337826 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:21:27.341332 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:27.344915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 930
I20260620 23:21:27.345144 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:21:27.345925 388 external_mini_cluster.cc:1428] Setting key 499f3c4aac05df6f565f520d9c7c5af7
I20260620 23:21:27.354161 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf: Bootstrap starting.
I20260620 23:21:27.357745 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:27.359042 997 log.cc:824] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:21:27 dist-test-slave-5kzz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:27.361869 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf: No bootstrap required, opened a new log
I20260620 23:21:27.368062 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:27.347884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.0.97.62:38875)
0620 23:21:27.348609 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:27.348620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:27.348658 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:21:27.350155 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:27.350175 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:27.350557 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:27.350927 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:27.352596 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:27.353852 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:27.355530 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:27.356187 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:27.356311 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:27.361060 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:27.361067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:27.361080 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:27.361114 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:27.364182 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:27.364926 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:27.364931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:27.364937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:27.365087 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:27.365449 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:27.365452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:27.365454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:27.365904 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:27.366119 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:27.366625 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 23:21:27.367100 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":353,"threads_started":1}
I20260620 23:21:27.370707 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe9796d9eb54195b7ff06d61ef181bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38875 } }
I20260620 23:21:27.371130 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:27.371235 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe9796d9eb54195b7ff06d61ef181bf, State: Initialized, Role: FOLLOWER
I20260620 23:21:27.371801 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe9796d9eb54195b7ff06d61ef181bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38875 } }
I20260620 23:21:27.372023 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:21:27.372169 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:21:27.372352 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:27.374661 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe9796d9eb54195b7ff06d61ef181bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38875 } }
I20260620 23:21:27.375187 997 leader_election.cc:302] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbe9796d9eb54195b7ff06d61ef181bf; no voters:
I20260620 23:21:27.375705 997 leader_election.cc:288] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:21:27.375833 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:27.376502 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [term 1 LEADER]: Becoming Leader. State: Replica: cbe9796d9eb54195b7ff06d61ef181bf, State: Running, Role: LEADER
I20260620 23:21:27.377039 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe9796d9eb54195b7ff06d61ef181bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38875 } }
I20260620 23:21:27.377668 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:21:27.379619 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbe9796d9eb54195b7ff06d61ef181bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe9796d9eb54195b7ff06d61ef181bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38875 } } }
I20260620 23:21:27.379762 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbe9796d9eb54195b7ff06d61ef181bf. Latest consensus state: current_term: 1 leader_uuid: "cbe9796d9eb54195b7ff06d61ef181bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe9796d9eb54195b7ff06d61ef181bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38875 } } }
I20260620 23:21:27.380100 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:27.380245 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:27.382140 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:21:27.388842 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:21:27.396800 1007 catalog_manager.cc:1347] Generated new cluster ID: e22a24c300a345f5bb2db96234342620
I20260620 23:21:27.396916 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:21:27.442600 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:21:27.443966 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:21:27.455561 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbe9796d9eb54195b7ff06d61ef181bf: Generated new TSK 0
I20260620 23:21:27.456357 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:21:27.495636 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38875
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:27.814065 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:27.814369 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:27.814529 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:27.822980 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:27.823123 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:27.823311 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:21:27.833091 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38875
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:27.834370 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:27.835711 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:27.848459 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:27.848510 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:27.849170 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:27.849188 1023 server_base.cc:1034] running on GCE node
I20260620 23:21:27.849871 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:27.850781 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:27.851972 1023 hybrid_clock.cc:648] HybridClock initialized: now 1781997687851929 us; error 45 us; skew 500 ppm
I20260620 23:21:27.852272 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:27 dist-test-slave-5kzz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:27.855296 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:21:27.856622 1023 webserver.cc:460] Webserver started at http://127.0.97.1:36917/ using document root <none> and password file <none>
I20260620 23:21:27.857270 1023 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:27.857415 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:27.857733 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:27.860690 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "287c01344f484b4c93b6488d4d051007"
format_stamp: "Formatted at 2026-06-20 23:21:27 on dist-test-slave-5kzz"
server_key: "f342150514c21bbea889cfb95acad18b"
server_key_iv: "39dbd873a27b3bbe72b8240ea17bb2dd"
server_key_version: "encryptionkey@0"
I20260620 23:21:27.861481 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "287c01344f484b4c93b6488d4d051007"
format_stamp: "Formatted at 2026-06-20 23:21:27 on dist-test-slave-5kzz"
server_key: "f342150514c21bbea889cfb95acad18b"
server_key_iv: "39dbd873a27b3bbe72b8240ea17bb2dd"
server_key_version: "encryptionkey@0"
I20260620 23:21:27.866333 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 23:21:27.869722 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:27.871031 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:21:27.871202 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "287c01344f484b4c93b6488d4d051007"
format_stamp: "Formatted at 2026-06-20 23:21:27 on dist-test-slave-5kzz"
server_key: "f342150514c21bbea889cfb95acad18b"
server_key_iv: "39dbd873a27b3bbe72b8240ea17bb2dd"
server_key_version: "encryptionkey@0"
I20260620 23:21:27.871414 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:27.898986 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:27.902966 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:27.903260 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:27.904227 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:27.905532 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:27.905633 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:27.905781 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:27.905859 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:27.937562 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45979
I20260620 23:21:27.937608 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45979 every 8 connection(s)
I20260620 23:21:27.938875 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:21:27.942895 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 1023
I20260620 23:21:27.943107 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:21:27.943727 388 external_mini_cluster.cc:1428] Setting key d9683f2f3ee8319482a3e59370e0fba1
Jun 20 23:21:27 dist-test-slave-5kzz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:27.959292 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:27.941295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36475 (local address 127.0.97.62:38875)
0620 23:21:27.941694 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:27.941699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:27.942740 (+ 1041us) server_negotiation.cc:407] Connection header received
0620 23:21:27.944653 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:27.944658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:27.944749 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:27.944875 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:27.946307 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:27.947049 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:27.948496 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:27.948869 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:27.949018 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:27.952536 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:27.952545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:27.952553 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:27.952596 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:27.956230 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:27.956926 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:27.956934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:27.956939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:27.957094 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:27.957525 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:27.957530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:27.957535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:27.957833 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:27.957993 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:27.958700 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 23:21:27.958923 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":299,"thread_start_us":208,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:21:27.960364 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:27.941896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38875 (local address 127.0.97.1:36475)
0620 23:21:27.942574 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:27.942609 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:27.944379 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:27.945121 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:27.945129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:27.945604 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:27.946145 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:27.946160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:27.947188 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:27.947197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:27.948358 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:27.948368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:27.949087 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:27.949092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:27.949280 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:27.949973 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:27.949994 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:27.952306 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:27.956377 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:27.956384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:27.956397 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:27.956802 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:27.957214 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:27.957217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:27.957219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:27.957399 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:27.957960 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:27.957968 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:27.958328 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 23:21:27.958597 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 23:21:27.959141 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":279,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:21:27.962145 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38875
I20260620 23:21:27.962458 1151 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:27.963229 1151 heartbeater.cc:511] Master 127.0.97.62:38875 requested a full tablet report, sending...
I20260620 23:21:27.965204 962 ts_manager.cc:194] Registered new tserver with Master: 287c01344f484b4c93b6488d4d051007 (127.0.97.1:45979)
I20260620 23:21:27.967355 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:21:27.981940 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38875
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:28.296528 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:28.296808 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:28.296901 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:28.304134 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:28.304308 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:28.304507 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:21:28.314805 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38875
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:28.316185 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:28.317541 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:28.330951 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:28.331032 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:28.331055 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:28.331910 1157 server_base.cc:1034] running on GCE node
I20260620 23:21:28.332585 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:28.333585 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:28.334775 1157 hybrid_clock.cc:648] HybridClock initialized: now 1781997688334725 us; error 38 us; skew 500 ppm
I20260620 23:21:28.335100 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:28 dist-test-slave-5kzz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:28.338528 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:21:28.340253 1157 webserver.cc:460] Webserver started at http://127.0.97.2:46085/ using document root <none> and password file <none>
I20260620 23:21:28.341078 1157 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:28.341217 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:28.341568 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:28.345520 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "385ae29c09214c0abeba272006644b42"
format_stamp: "Formatted at 2026-06-20 23:21:28 on dist-test-slave-5kzz"
server_key: "2d4d4af9363bdbde5900a4fda444a738"
server_key_iv: "4cbfc72896c315762ddc5a9daeab086b"
server_key_version: "encryptionkey@0"
I20260620 23:21:28.346489 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "385ae29c09214c0abeba272006644b42"
format_stamp: "Formatted at 2026-06-20 23:21:28 on dist-test-slave-5kzz"
server_key: "2d4d4af9363bdbde5900a4fda444a738"
server_key_iv: "4cbfc72896c315762ddc5a9daeab086b"
server_key_version: "encryptionkey@0"
I20260620 23:21:28.353139 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 23:21:28.357398 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:28.358812 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:21:28.358968 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "385ae29c09214c0abeba272006644b42"
format_stamp: "Formatted at 2026-06-20 23:21:28 on dist-test-slave-5kzz"
server_key: "2d4d4af9363bdbde5900a4fda444a738"
server_key_iv: "4cbfc72896c315762ddc5a9daeab086b"
server_key_version: "encryptionkey@0"
I20260620 23:21:28.359170 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:28.369910 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:28.374132 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:28.374449 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:28.375479 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:28.376796 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:28.376865 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:28.376976 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:28.377044 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:28.409008 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40767
I20260620 23:21:28.409046 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40767 every 8 connection(s)
I20260620 23:21:28.410276 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:21:28.417891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 1157
I20260620 23:21:28.418087 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:21:28.418499 388 external_mini_cluster.cc:1428] Setting key 076760d31c11f1f4732a8ed78e6e8d12
Jun 20 23:21:28 dist-test-slave-5kzz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:28.432430 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:28.412867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58703 (local address 127.0.97.62:38875)
0620 23:21:28.413088 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:28.413094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:28.414282 (+ 1188us) server_negotiation.cc:407] Connection header received
0620 23:21:28.416039 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:28.416043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:28.416106 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:28.416195 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:28.417660 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.418463 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.420338 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.420777 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.420870 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:28.425334 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:28.425342 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:28.425350 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:28.425394 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:28.429149 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:28.429884 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:28.429892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:28.429899 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:28.430055 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:28.430488 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:28.430496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:28.430502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:28.430908 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:28.431055 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:28.431771 (+ 716us) server_negotiation.cc:299] Negotiation successful
0620 23:21:28.432026 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:21:28.433228 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:28.413441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38875 (local address 127.0.97.2:58703)
0620 23:21:28.414113 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:28.414156 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:28.415797 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:28.416406 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:28.416414 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:28.416896 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:28.417443 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:28.417458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.418618 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.418624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:28.420209 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:28.420220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.420994 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.421001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:28.421339 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:28.422083 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:28.422117 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:28.425103 (+ 2986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:28.429294 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:28.429302 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:28.429316 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:28.429741 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:28.430160 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:28.430163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:28.430165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:28.430345 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:28.431023 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:28.431029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:28.431369 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 23:21:28.431631 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 23:21:28.432062 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":273,"threads_started":1}
I20260620 23:21:28.434700 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38875
I20260620 23:21:28.434998 1285 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:21:28.435739 1285 heartbeater.cc:511] Master 127.0.97.62:38875 requested a full tablet report, sending...
I20260620 23:21:28.437136 962 ts_manager.cc:194] Registered new tserver with Master: 385ae29c09214c0abeba272006644b42 (127.0.97.2:40767)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 23:21:28.438148 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:21:28.458230 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38875
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:28.770236 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:28.770507 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:28.770602 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:28.777928 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:28.778075 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:28.778280 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:21:28.788017 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38875
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:28.789237 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:28.790526 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:28.803296 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:28.803390 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:28.803409 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:28.804171 1290 server_base.cc:1034] running on GCE node
I20260620 23:21:28.804807 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:28.805794 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:28.807008 1290 hybrid_clock.cc:648] HybridClock initialized: now 1781997688806966 us; error 60 us; skew 500 ppm
I20260620 23:21:28.807320 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:28 dist-test-slave-5kzz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:28.810285 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:21:28.811779 1290 webserver.cc:460] Webserver started at http://127.0.97.3:38391/ using document root <none> and password file <none>
I20260620 23:21:28.812505 1290 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:28.812646 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:28.812963 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:28.815793 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "7ffdd816d303474fad9490d2b1717005"
format_stamp: "Formatted at 2026-06-20 23:21:28 on dist-test-slave-5kzz"
server_key: "39c847bb9f3f7d43e85460710df93576"
server_key_iv: "923022b3316e3f69ffcb49e5208cca66"
server_key_version: "encryptionkey@0"
I20260620 23:21:28.816531 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7ffdd816d303474fad9490d2b1717005"
format_stamp: "Formatted at 2026-06-20 23:21:28 on dist-test-slave-5kzz"
server_key: "39c847bb9f3f7d43e85460710df93576"
server_key_iv: "923022b3316e3f69ffcb49e5208cca66"
server_key_version: "encryptionkey@0"
I20260620 23:21:28.821899 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:21:28.825737 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:28.827164 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:21:28.827312 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "7ffdd816d303474fad9490d2b1717005"
format_stamp: "Formatted at 2026-06-20 23:21:28 on dist-test-slave-5kzz"
server_key: "39c847bb9f3f7d43e85460710df93576"
server_key_iv: "923022b3316e3f69ffcb49e5208cca66"
server_key_version: "encryptionkey@0"
I20260620 23:21:28.827494 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:28.848102 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:28.851857 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:28.852079 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:28.852981 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:28.854209 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:28.854275 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:28.854360 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:28.854401 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:28.886642 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36021
I20260620 23:21:28.886708 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36021 every 8 connection(s)
I20260620 23:21:28.887946 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:21:28.893980 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 1290
I20260620 23:21:28.894182 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:21:28.894608 388 external_mini_cluster.cc:1428] Setting key 13e26d91b5155769c27e4a5b27d31f5c
Jun 20 23:21:28 dist-test-slave-5kzz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:28.907548 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:28.890367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44257 (local address 127.0.97.62:38875)
0620 23:21:28.890536 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:28.890541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:28.891767 (+ 1226us) server_negotiation.cc:407] Connection header received
0620 23:21:28.893896 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:28.893903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:28.893996 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:28.894122 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:28.896030 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.896774 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.898302 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.898765 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.898902 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:28.902099 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:28.902110 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:28.902113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:28.902143 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:28.904424 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:28.905282 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:28.905289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:28.905298 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:28.905421 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:28.905833 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:28.905840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:28.905843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:28.906112 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:28.906264 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:28.907130 (+ 866us) server_negotiation.cc:299] Negotiation successful
0620 23:21:28.907305 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:21:28.908614 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:28.890890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38875 (local address 127.0.97.3:44257)
0620 23:21:28.891600 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:28.891644 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:28.893611 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:28.894391 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:28.894399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:28.895138 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:28.895842 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:28.895859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.896940 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.896944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:28.898149 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:28.898160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.899002 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.899009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:28.899233 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:28.899826 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:28.899852 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:28.901895 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:28.904563 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:28.904572 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:28.904589 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:28.905147 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:28.905529 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:28.905535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:28.905538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:28.905722 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:28.906236 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:28.906245 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:28.906646 (+ 401us) client_negotiation.cc:769] Sending connection context
0620 23:21:28.907034 (+ 388us) client_negotiation.cc:240] Negotiation successful
0620 23:21:28.907422 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":266,"threads_started":1}
I20260620 23:21:28.910081 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38875
I20260620 23:21:28.910386 1418 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:28.911128 1418 heartbeater.cc:511] Master 127.0.97.62:38875 requested a full tablet report, sending...
I20260620 23:21:28.912482 961 ts_manager.cc:194] Registered new tserver with Master: 7ffdd816d303474fad9490d2b1717005 (127.0.97.3:36021)
I20260620 23:21:28.913303 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44257
I20260620 23:21:28.922235 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:21:28.941680 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:28.928518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.0.97.62:38875)
0620 23:21:28.928755 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:28.928764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:28.929122 (+ 358us) server_negotiation.cc:407] Connection header received
0620 23:21:28.929251 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:28.929256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:28.929334 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:28.929425 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:28.930863 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.931661 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.932957 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.933346 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.933445 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:28.935688 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:28.935692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:28.935695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:28.935725 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:28.938798 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:28.939373 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:28.939378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:28.939382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:28.939515 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:28.939840 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:28.939844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:28.939848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:28.940233 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:28.940482 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:28.941173 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 23:21:28.941373 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:21:28.948064 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:21:28.954654 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:21:28.970319 1151 heartbeater.cc:503] Master 127.0.97.62:38875 was elected leader, sending a full tablet report...
I20260620 23:21:28.990897 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:28.966851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.0.97.3:36021)
0620 23:21:28.983603 (+ 16752us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:28.983613 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:28.983652 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:21:28.983742 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:28.983751 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:28.983981 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:28.984163 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:28.984765 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.986498 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.988829 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.989765 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.989908 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:28.990142 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:28.990368 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 23:21:28.990522 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16570,"thread_start_us":16347,"threads_started":1}
I20260620 23:21:28.991003 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:28.966546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36021 (local address 127.0.0.1:43324)
0620 23:21:28.979524 (+ 12978us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:28.979600 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:28.979845 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:28.984171 (+ 4326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:28.984177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:28.984206 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:28.984556 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:28.984569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.986634 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.986639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:28.988635 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:28.988646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.989948 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.989954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:28.990131 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:28.990183 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:21:28.990336 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 23:21:28.990513 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12647,"mutex_wait_us":159,"thread_start_us":14890,"threads_started":1}
I20260620 23:21:28.991801 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:28.966769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45979 (local address 127.0.0.1:53206)
0620 23:21:28.981148 (+ 14379us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:28.981176 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:28.981324 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:28.982985 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:28.982993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:28.983028 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:28.983804 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:28.983818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.986103 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.986109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:28.988710 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:28.988724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.990825 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.990830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:28.991054 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:28.991168 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 23:21:28.991261 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:21:28.991406 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14203,"mutex_wait_us":179,"thread_start_us":13294,"threads_started":1}
I20260620 23:21:28.991801 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:28.967875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.0.97.2:40767)
0620 23:21:28.981113 (+ 13238us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:28.981127 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:28.981721 (+ 594us) server_negotiation.cc:407] Connection header received
0620 23:21:28.981884 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:28.981894 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:28.982193 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:28.983493 (+ 1300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:28.983978 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.985695 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.988892 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.990295 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.990414 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:28.990590 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:28.991374 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 23:21:28.991539 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13082,"thread_start_us":12986,"threads_started":1}
I20260620 23:21:28.991882 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:28.967266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40767 (local address 127.0.0.1:37160)
0620 23:21:28.981580 (+ 14314us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:28.981608 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:28.981747 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:28.982448 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:28.982455 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:28.982485 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:28.983707 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:28.983719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.985818 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.985826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:28.988710 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:28.988724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.990806 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.990814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:28.991060 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:28.991169 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 23:21:28.991251 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 23:21:28.991483 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14197,"mutex_wait_us":387,"thread_start_us":14037,"threads_started":1}
I20260620 23:21:28.992453 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:28.967507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.0.97.1:45979)
0620 23:21:28.982324 (+ 14817us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:28.982334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:28.982362 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:21:28.982451 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:28.982458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:28.982707 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:28.982908 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:28.983984 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.985974 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.988888 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:28.990288 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:28.990414 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:28.990624 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:28.991992 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0620 23:21:28.992164 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14637,"thread_start_us":14485,"threads_started":1}
I20260620 23:21:28.995174 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2b0d3b44df4df3b2d29350f725ec85 (DEFAULT_TABLE table=test-table [id=f5091acddbea423fac68dabc8a5994c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:28.995699 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2b0d3b44df4df3b2d29350f725ec85 (DEFAULT_TABLE table=test-table [id=f5091acddbea423fac68dabc8a5994c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:28.996452 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2b0d3b44df4df3b2d29350f725ec85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:28.995702 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2b0d3b44df4df3b2d29350f725ec85 (DEFAULT_TABLE table=test-table [id=f5091acddbea423fac68dabc8a5994c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:28.997473 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2b0d3b44df4df3b2d29350f725ec85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:28.997818 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2b0d3b44df4df3b2d29350f725ec85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:29.008023 1436 tablet_bootstrap.cc:492] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005: Bootstrap starting.
I20260620 23:21:29.011250 1436 tablet_bootstrap.cc:654] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:29.012451 1436 log.cc:824] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:29.013008 1438 tablet_bootstrap.cc:492] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007: Bootstrap starting.
I20260620 23:21:29.015395 1437 tablet_bootstrap.cc:492] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42: Bootstrap starting.
I20260620 23:21:29.017974 1438 tablet_bootstrap.cc:654] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:29.018028 1437 tablet_bootstrap.cc:654] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:29.019366 1437 log.cc:824] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:29.019404 1438 log.cc:824] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:29.021651 1436 tablet_bootstrap.cc:492] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005: No bootstrap required, opened a new log
I20260620 23:21:29.021934 1436 ts_tablet_manager.cc:1397] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260620 23:21:29.023113 1438 tablet_bootstrap.cc:492] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007: No bootstrap required, opened a new log
I20260620 23:21:29.023363 1438 ts_tablet_manager.cc:1397] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20260620 23:21:29.023591 1437 tablet_bootstrap.cc:492] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42: No bootstrap required, opened a new log
I20260620 23:21:29.023821 1437 ts_tablet_manager.cc:1397] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 23:21:29.031167 1438 raft_consensus.cc:348] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ffdd816d303474fad9490d2b1717005" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36021 } } peers { permanent_uuid: "287c01344f484b4c93b6488d4d051007" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45979 } } peers { permanent_uuid: "385ae29c09214c0abeba272006644b42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40767 } }
I20260620 23:21:29.031723 1438 raft_consensus.cc:374] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:29.031852 1438 raft_consensus.cc:729] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287c01344f484b4c93b6488d4d051007, State: Initialized, Role: FOLLOWER
I20260620 23:21:29.032469 1438 consensus_queue.cc:260] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ffdd816d303474fad9490d2b1717005" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36021 } } peers { permanent_uuid: "287c01344f484b4c93b6488d4d051007" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45979 } } peers { permanent_uuid: "385ae29c09214c0abeba272006644b42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40767 } }
I20260620 23:21:29.032342 1436 raft_consensus.cc:348] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ffdd816d303474fad9490d2b1717005" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36021 } } peers { permanent_uuid: "287c01344f484b4c93b6488d4d051007" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45979 } } peers { permanent_uuid: "385ae29c09214c0abeba272006644b42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40767 } }
I20260620 23:21:29.032723 1436 raft_consensus.cc:374] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:29.032841 1436 raft_consensus.cc:729] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffdd816d303474fad9490d2b1717005, State: Initialized, Role: FOLLOWER
I20260620 23:21:29.033443 1436 consensus_queue.cc:260] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ffdd816d303474fad9490d2b1717005" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36021 } } peers { permanent_uuid: "287c01344f484b4c93b6488d4d051007" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45979 } } peers { permanent_uuid: "385ae29c09214c0abeba272006644b42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40767 } }
I20260620 23:21:29.034358 1438 ts_tablet_manager.cc:1428] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260620 23:21:29.034335 1437 raft_consensus.cc:348] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ffdd816d303474fad9490d2b1717005" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36021 } } peers { permanent_uuid: "287c01344f484b4c93b6488d4d051007" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45979 } } peers { permanent_uuid: "385ae29c09214c0abeba272006644b42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40767 } }
I20260620 23:21:29.034957 1437 raft_consensus.cc:374] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:29.035085 1437 raft_consensus.cc:729] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 385ae29c09214c0abeba272006644b42, State: Initialized, Role: FOLLOWER
I20260620 23:21:29.035413 1418 heartbeater.cc:503] Master 127.0.97.62:38875 was elected leader, sending a full tablet report...
I20260620 23:21:29.035753 1437 consensus_queue.cc:260] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ffdd816d303474fad9490d2b1717005" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36021 } } peers { permanent_uuid: "287c01344f484b4c93b6488d4d051007" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45979 } } peers { permanent_uuid: "385ae29c09214c0abeba272006644b42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40767 } }
I20260620 23:21:29.036026 1436 ts_tablet_manager.cc:1428] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 23:21:29.037582 1437 ts_tablet_manager.cc:1428] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 23:21:29.038085 1285 heartbeater.cc:503] Master 127.0.97.62:38875 was elected leader, sending a full tablet report...
W20260620 23:21:29.139891 1419 tablet.cc:2369] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:21:29.162433 1286 tablet.cc:2369] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:21:29.170053 1442 raft_consensus.cc:482] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:21:29.170382 1442 raft_consensus.cc:504] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ffdd816d303474fad9490d2b1717005" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36021 } } peers { permanent_uuid: "287c01344f484b4c93b6488d4d051007" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45979 } } peers { permanent_uuid: "385ae29c09214c0abeba272006644b42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40767 } }
I20260620 23:21:29.172070 1442 leader_election.cc:288] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ffdd816d303474fad9490d2b1717005 (127.0.97.3:36021), 385ae29c09214c0abeba272006644b42 (127.0.97.2:40767)
I20260620 23:21:29.179399 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:29.172598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48165 (local address 127.0.97.2:40767)
0620 23:21:29.172846 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:29.172855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:29.173695 (+ 840us) server_negotiation.cc:407] Connection header received
0620 23:21:29.173871 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:29.173875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:29.173929 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:29.174031 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:29.174722 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.175424 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.177233 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.178192 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.178288 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:29.178402 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:29.178983 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 23:21:29.179123 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:21:29.179476 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:29.172498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40767 (local address 127.0.97.1:48165)
0620 23:21:29.173564 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:29.173579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:29.173724 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:29.174063 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:29.174067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:29.174091 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:29.174524 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:29.174537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.175545 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.175553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:29.177081 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:29.177092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.178580 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.178586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:29.178777 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:29.178823 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 23:21:29.178945 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 23:21:29.179141 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":22,"thread_start_us":714,"threads_started":1}
I20260620 23:21:29.179792 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:29.172358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39577 (local address 127.0.97.3:36021)
0620 23:21:29.172558 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:29.172564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:29.173641 (+ 1077us) server_negotiation.cc:407] Connection header received
0620 23:21:29.173824 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:29.173829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:29.173904 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:29.173974 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:29.174560 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.175468 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.177393 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.178860 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.178969 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:29.179062 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:29.179497 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 23:21:29.179622 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:21:29.179957 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:29.172455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36021 (local address 127.0.97.1:39577)
0620 23:21:29.173531 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:29.173547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:29.173684 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:29.174024 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:29.174028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:29.174050 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:29.174429 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:29.174441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.175652 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.175659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:29.177250 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:29.177262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.179136 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.179142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:29.179310 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:29.179342 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:21:29.179439 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:21:29.179593 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":17,"thread_start_us":550,"threads_started":1}
I20260620 23:21:29.180721 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2b0d3b44df4df3b2d29350f725ec85" candidate_uuid: "287c01344f484b4c93b6488d4d051007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "385ae29c09214c0abeba272006644b42" is_pre_election: true
I20260620 23:21:29.180835 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2b0d3b44df4df3b2d29350f725ec85" candidate_uuid: "287c01344f484b4c93b6488d4d051007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ffdd816d303474fad9490d2b1717005" is_pre_election: true
I20260620 23:21:29.181245 1240 raft_consensus.cc:2454] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287c01344f484b4c93b6488d4d051007 in term 0.
I20260620 23:21:29.181310 1373 raft_consensus.cc:2454] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287c01344f484b4c93b6488d4d051007 in term 0.
I20260620 23:21:29.181877 1041 leader_election.cc:302] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287c01344f484b4c93b6488d4d051007, 385ae29c09214c0abeba272006644b42; no voters:
I20260620 23:21:29.182255 1442 raft_consensus.cc:2781] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:21:29.182412 1442 raft_consensus.cc:482] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:21:29.182513 1442 raft_consensus.cc:3037] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:29.184839 1442 raft_consensus.cc:504] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ffdd816d303474fad9490d2b1717005" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36021 } } peers { permanent_uuid: "287c01344f484b4c93b6488d4d051007" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45979 } } peers { permanent_uuid: "385ae29c09214c0abeba272006644b42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40767 } }
I20260620 23:21:29.185437 1442 leader_election.cc:288] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [CANDIDATE]: Term 1 election: Requested vote from peers 7ffdd816d303474fad9490d2b1717005 (127.0.97.3:36021), 385ae29c09214c0abeba272006644b42 (127.0.97.2:40767)
I20260620 23:21:29.185789 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2b0d3b44df4df3b2d29350f725ec85" candidate_uuid: "287c01344f484b4c93b6488d4d051007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ffdd816d303474fad9490d2b1717005"
I20260620 23:21:29.185854 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2b0d3b44df4df3b2d29350f725ec85" candidate_uuid: "287c01344f484b4c93b6488d4d051007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "385ae29c09214c0abeba272006644b42"
I20260620 23:21:29.186010 1373 raft_consensus.cc:3037] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:29.186033 1240 raft_consensus.cc:3037] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:29.189105 1373 raft_consensus.cc:2454] T 9c2b0d3b44df4df3b2d29350f725ec85 P 7ffdd816d303474fad9490d2b1717005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287c01344f484b4c93b6488d4d051007 in term 1.
I20260620 23:21:29.189105 1240 raft_consensus.cc:2454] T 9c2b0d3b44df4df3b2d29350f725ec85 P 385ae29c09214c0abeba272006644b42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287c01344f484b4c93b6488d4d051007 in term 1.
I20260620 23:21:29.189517 1039 leader_election.cc:302] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287c01344f484b4c93b6488d4d051007, 7ffdd816d303474fad9490d2b1717005; no voters:
I20260620 23:21:29.189786 1442 raft_consensus.cc:2781] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:29.190222 1442 raft_consensus.cc:686] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [term 1 LEADER]: Becoming Leader. State: Replica: 287c01344f484b4c93b6488d4d051007, State: Running, Role: LEADER
I20260620 23:21:29.190668 1442 consensus_queue.cc:237] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ffdd816d303474fad9490d2b1717005" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36021 } } peers { permanent_uuid: "287c01344f484b4c93b6488d4d051007" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45979 } } peers { permanent_uuid: "385ae29c09214c0abeba272006644b42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40767 } }
W20260620 23:21:29.191283 1152 tablet.cc:2369] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:21:29.195761 960 catalog_manager.cc:5495] T 9c2b0d3b44df4df3b2d29350f725ec85 P 287c01344f484b4c93b6488d4d051007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 287c01344f484b4c93b6488d4d051007 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "287c01344f484b4c93b6488d4d051007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ffdd816d303474fad9490d2b1717005" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287c01344f484b4c93b6488d4d051007" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "385ae29c09214c0abeba272006644b42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40767 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:21:29 dist-test-slave-5kzz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:21:29.232988 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:29.218362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53220 (local address 127.0.97.1:45979)
0620 23:21:29.218577 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:29.218586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:29.219040 (+ 454us) server_negotiation.cc:407] Connection header received
0620 23:21:29.219232 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:29.219241 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:29.219300 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:29.219410 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:29.220823 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.221526 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.222805 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.223192 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.223324 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:29.227185 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:29.227199 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:29.227214 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:29.227244 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:29.230236 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:29.230995 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:29.231002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:29.231004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:29.231163 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:29.231551 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:29.231556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:29.231559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:29.231883 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:29.232175 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:29.232452 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 23:21:29.232665 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 23:21:29 dist-test-slave-5kzz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997689, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:21:29 dist-test-slave-5kzz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997689, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:21:29.276434 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:29.263474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.0.97.1:45979)
0620 23:21:29.263673 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:29.263683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:29.264029 (+ 346us) server_negotiation.cc:407] Connection header received
0620 23:21:29.264163 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:29.264167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:29.264227 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:29.264311 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:29.265559 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.266233 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.267570 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.267974 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.268059 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:29.271536 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:29.271541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:29.271544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:29.271572 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:29.273794 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:29.274343 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:29.274346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:29.274348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:29.274446 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:29.274901 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:29.274904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:29.274905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:29.275539 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:29.275821 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:29.275983 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 23:21:29.276135 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 23:21:29 dist-test-slave-5kzz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:21:29 dist-test-slave-5kzz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:21:29.317831 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:29.305024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.0.97.1:45979)
0620 23:21:29.305205 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:29.305213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:29.305497 (+ 284us) server_negotiation.cc:407] Connection header received
0620 23:21:29.305700 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:29.305703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:29.305777 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:29.305884 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:29.307218 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.307947 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.309134 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.309518 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.309646 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:29.313201 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:29.313205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:29.313208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:29.313238 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:29.315518 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:29.316103 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:29.316109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:29.316112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:29.316227 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:29.316551 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:29.316557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:29.316561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:29.316816 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:29.317062 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:29.317316 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 23:21:29.317518 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 23:21:29.319496 1066 tablet_service.cc:2908] Rejecting scan request for tablet 9c2b0d3b44df4df3b2d29350f725ec85: Uninitialized: clean time has not yet been initialized
I20260620 23:21:29.322520 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 1023
I20260620 23:21:29.334827 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:29.323295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.0.97.1:45979)
0620 23:21:29.323437 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:29.323440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:29.323757 (+ 317us) server_negotiation.cc:407] Connection header received
0620 23:21:29.323943 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:29.323948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:29.324012 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:29.324116 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:29.325190 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.325956 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.327190 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.327582 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.327727 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:29.330155 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:29.330161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:29.330165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:29.330200 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:29.332826 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:29.333377 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:29.333381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:29.333383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:29.333485 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:29.333770 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:29.333773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:29.333775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:29.333985 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:29.334141 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:29.334339 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 23:21:29.334502 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:21:29.335436 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:29.431565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 1023
I20260620 23:21:29.448609 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 1157
Jun 20 23:21:29 dist-test-slave-5kzz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:21:29.462883 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:29.449444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.0.97.2:40767)
0620 23:21:29.449610 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:29.449615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:29.449632 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:21:29.449688 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:29.449692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:29.449780 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:29.449860 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:29.451018 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.451812 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.453046 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.453436 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.453531 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:29.457427 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:29.457438 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:29.457453 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:29.457482 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:29.460128 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:29.460801 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:29.460808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:29.460810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:29.460935 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:29.461312 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:29.461319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:29.461322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:29.461670 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:29.461985 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:29.462255 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:21:29.462465 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:21:29.463536 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:29.549472 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 1157
I20260620 23:21:29.566004 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 1290
Jun 20 23:21:29 dist-test-slave-5kzz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:21:29.582113 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:29.566783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.0.97.3:36021)
0620 23:21:29.566953 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:29.566962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:29.566997 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:21:29.567189 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:29.567197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:29.567272 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:29.567379 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:29.568521 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.569222 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.570437 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:29.571014 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:29.571149 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:29.575607 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:29.575627 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:29.575651 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:29.575695 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:29.579570 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:29.580342 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:29.580368 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:29.580372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:29.580530 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:29.580991 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:29.580995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:29.580997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:29.581299 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:29.581497 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:29.581704 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 23:21:29.581877 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:21:29.582758 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:29.667455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 1290
I20260620 23:21:29.683475 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 930
I20260620 23:21:29.684607 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:29.787971 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 930
2026-06-20T23:21:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4098 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:21:29 dist-test-slave-5kzz krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:21:29 dist-test-slave-5kzz krb5kdc[1466](info): set up 2 sockets
Jun 20 23:21:29 dist-test-slave-5kzz krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260620 23:21:30.866564 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:21:30 dist-test-slave-5kzz krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:21:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:21:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:21:31.031685 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35187
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:31.342280 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:31.342593 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:31.342710 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:31.349988 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:21:31.350097 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:21:31.350137 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:31.350190 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:31.350239 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:21:31.350293 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:21:31.360139 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35187
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:31.361337 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:31.362573 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:31.373687 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:31.373729 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:31.373767 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:31.374554 1479 server_base.cc:1034] running on GCE node
I20260620 23:21:31.375355 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:31.376695 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:31.377945 1479 hybrid_clock.cc:648] HybridClock initialized: now 1781997691377896 us; error 47 us; skew 500 ppm
I20260620 23:21:31.378288 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:31 dist-test-slave-5kzz krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:31.381237 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:21:31.382652 1479 webserver.cc:460] Webserver started at http://127.0.97.62:44347/ using document root <none> and password file <none>
I20260620 23:21:31.383404 1479 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:31.383535 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:31.383855 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:31.386770 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "0e74e4ab23a146a0933a2ea74891f0cd"
format_stamp: "Formatted at 2026-06-20 23:21:31 on dist-test-slave-5kzz"
server_key: "fec5b691d18a3902f1b0f1ca8a9dcb53"
server_key_iv: "ae88a55ff3296e26c3dda97cb67db6b1"
server_key_version: "encryptionkey@0"
I20260620 23:21:31.387554 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "0e74e4ab23a146a0933a2ea74891f0cd"
format_stamp: "Formatted at 2026-06-20 23:21:31 on dist-test-slave-5kzz"
server_key: "fec5b691d18a3902f1b0f1ca8a9dcb53"
server_key_iv: "ae88a55ff3296e26c3dda97cb67db6b1"
server_key_version: "encryptionkey@0"
I20260620 23:21:31.392313 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:21:31.395660 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:31.396906 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:21:31.397086 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "0e74e4ab23a146a0933a2ea74891f0cd"
format_stamp: "Formatted at 2026-06-20 23:21:31 on dist-test-slave-5kzz"
server_key: "fec5b691d18a3902f1b0f1ca8a9dcb53"
server_key_iv: "ae88a55ff3296e26c3dda97cb67db6b1"
server_key_version: "encryptionkey@0"
I20260620 23:21:31.397286 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:31.422127 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:31.426021 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:31.426295 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:31.446818 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35187
I20260620 23:21:31.446854 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35187 every 8 connection(s)
I20260620 23:21:31.448166 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:21:31.451781 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:31.457700 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 1479
I20260620 23:21:31.457922 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:21:31.458576 388 external_mini_cluster.cc:1428] Setting key d4ef9cbbfba01328db9adbe0a0b7e179
I20260620 23:21:31.462183 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd: Bootstrap starting.
I20260620 23:21:31.465909 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:31.467011 1556 log.cc:824] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:31.470031 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd: No bootstrap required, opened a new log
Jun 20 23:21:31 dist-test-slave-5kzz krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:31.477691 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e74e4ab23a146a0933a2ea74891f0cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35187 } }
I20260620 23:21:31.478078 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:31.478224 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e74e4ab23a146a0933a2ea74891f0cd, State: Initialized, Role: FOLLOWER
I20260620 23:21:31.478946 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e74e4ab23a146a0933a2ea74891f0cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35187 } }
I20260620 23:21:31.479224 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:21:31.479369 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:21:31.479611 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:31.480700 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:31.460556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.0.97.62:35187)
0620 23:21:31.461336 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:31.461347 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:31.461388 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:21:31.463172 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:31.463194 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:31.463588 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:31.463931 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:31.465638 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:31.467055 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:31.468663 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:31.469311 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:31.469437 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:31.473436 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:31.473444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:31.473458 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:31.473496 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:31.476547 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:31.477268 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:31.477277 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:31.477292 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:31.477452 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:31.477814 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:31.477820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:31.477823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:31.478492 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:31.478814 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:31.479225 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 23:21:31.479653 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":376,"threads_started":1}
I20260620 23:21:31.482363 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e74e4ab23a146a0933a2ea74891f0cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35187 } }
I20260620 23:21:31.482959 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e74e4ab23a146a0933a2ea74891f0cd; no voters:
I20260620 23:21:31.483558 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:21:31.483659 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:31.484334 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [term 1 LEADER]: Becoming Leader. State: Replica: 0e74e4ab23a146a0933a2ea74891f0cd, State: Running, Role: LEADER
I20260620 23:21:31.484897 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e74e4ab23a146a0933a2ea74891f0cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35187 } }
I20260620 23:21:31.485641 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:21:31.487818 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e74e4ab23a146a0933a2ea74891f0cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e74e4ab23a146a0933a2ea74891f0cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35187 } } }
I20260620 23:21:31.488277 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:31.489045 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e74e4ab23a146a0933a2ea74891f0cd. Latest consensus state: current_term: 1 leader_uuid: "0e74e4ab23a146a0933a2ea74891f0cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e74e4ab23a146a0933a2ea74891f0cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35187 } } }
I20260620 23:21:31.489372 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:31.489954 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:21:31.495029 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:21:31.501680 1578 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:21:31.504890 1569 catalog_manager.cc:1347] Generated new cluster ID: ccf94c56b93f42f78c9943313c6343fa
I20260620 23:21:31.505007 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:21:31.525123 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:21:31.526535 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:21:31.537297 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd: Generated new TSK 0
I20260620 23:21:31.538139 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:21:31.588291 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35187
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:31.922035 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:31.922297 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:31.922390 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:31.929522 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:31.929626 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:31.929790 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:21:31.929904 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:21:31.940428 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:31.941784 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:31.943713 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:31.956796 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:31.956741 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:31.956915 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:31.963095 1582 server_base.cc:1034] running on GCE node
I20260620 23:21:31.963688 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:31.964564 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:31.965790 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781997691965726 us; error 37 us; skew 500 ppm
I20260620 23:21:31.966086 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:31 dist-test-slave-5kzz krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:31.969197 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:21:31.970665 1582 webserver.cc:460] Webserver started at http://127.0.97.1:34237/ using document root <none> and password file <none>
I20260620 23:21:31.971450 1582 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:31.971586 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:31.971905 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:31.974833 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "619207749c5143e9add13c4f3551c362"
format_stamp: "Formatted at 2026-06-20 23:21:31 on dist-test-slave-5kzz"
server_key: "8a846bee2453f14ff065d6a3a5db55cb"
server_key_iv: "338e7d3c7fc811db4cb2180eebe30b23"
server_key_version: "encryptionkey@0"
I20260620 23:21:31.975625 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "619207749c5143e9add13c4f3551c362"
format_stamp: "Formatted at 2026-06-20 23:21:31 on dist-test-slave-5kzz"
server_key: "8a846bee2453f14ff065d6a3a5db55cb"
server_key_iv: "338e7d3c7fc811db4cb2180eebe30b23"
server_key_version: "encryptionkey@0"
I20260620 23:21:31.980690 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:21:31.984018 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:31.985284 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:21:31.985451 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "619207749c5143e9add13c4f3551c362"
format_stamp: "Formatted at 2026-06-20 23:21:31 on dist-test-slave-5kzz"
server_key: "8a846bee2453f14ff065d6a3a5db55cb"
server_key_iv: "338e7d3c7fc811db4cb2180eebe30b23"
server_key_version: "encryptionkey@0"
I20260620 23:21:31.985656 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:31.999162 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:32.003448 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:32.003717 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:32.006165 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:32.006247 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:32.006390 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:32.006459 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:21:32 dist-test-slave-5kzz krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:32.029152 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:32.007164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50793 (local address 127.0.97.62:35187)
0620 23:21:32.007584 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:32.007588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:32.008562 (+ 974us) server_negotiation.cc:407] Connection header received
0620 23:21:32.010413 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:32.010419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:32.010506 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:32.010620 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:32.012681 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.013630 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.016925 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.017500 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.017668 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:32.022594 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:32.022600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:32.022606 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:32.022642 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:32.025439 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:32.026361 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:32.026366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:32.026369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:32.026480 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:32.027139 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:32.027145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:32.027148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:32.027550 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:32.027717 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:32.028599 (+ 882us) server_negotiation.cc:299] Negotiation successful
0620 23:21:32.028798 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":251,"threads_started":1}
I20260620 23:21:32.030594 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:32.007635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35187 (local address 127.0.97.1:50793)
0620 23:21:32.008425 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:32.008463 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:32.010166 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:32.010912 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:32.010928 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:32.011724 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:32.012523 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:32.012540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.013849 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.013856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:32.016770 (+ 2914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:32.016784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.017994 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.018002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:32.018394 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:32.019189 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:32.019225 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:32.022361 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:32.025575 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:32.025585 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:32.025605 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:32.026196 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:32.026635 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:32.026642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:32.026647 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:32.027033 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:32.027695 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:32.027705 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:32.028156 (+ 451us) client_negotiation.cc:769] Sending connection context
0620 23:21:32.028524 (+ 368us) client_negotiation.cc:240] Negotiation successful
0620 23:21:32.029071 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":337,"threads_started":1}
I20260620 23:21:32.048905 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41781
I20260620 23:21:32.048911 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41781 every 8 connection(s)
I20260620 23:21:32.050351 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:21:32.052853 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35187
I20260620 23:21:32.053293 1718 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:32.054072 1718 heartbeater.cc:511] Master 127.0.97.62:35187 requested a full tablet report, sending...
I20260620 23:21:32.056226 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:32.037675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59389 (local address 127.0.97.62:35187)
0620 23:21:32.037905 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:32.037915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:32.038460 (+ 545us) server_negotiation.cc:407] Connection header received
0620 23:21:32.038586 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:32.038594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:32.038716 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:32.038835 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:32.040744 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.041614 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.043949 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.044640 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.044767 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:32.050365 (+ 5598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:32.050377 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:32.050381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:32.050436 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:32.053659 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:32.054365 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:32.054372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:32.054375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:32.054524 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:32.054928 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:32.054933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:32.054936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:32.055273 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:32.055391 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:32.055834 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 23:21:32.055989 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:21:32.056401 1511 ts_manager.cc:194] Registered new tserver with Master: 619207749c5143e9add13c4f3551c362 (127.0.97.1:41781)
I20260620 23:21:32.056470 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:32.037697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35187 (local address 127.0.97.1:59389)
0620 23:21:32.038369 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:32.038384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:32.038490 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:32.039029 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:32.039037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:32.039987 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:32.040601 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:32.040620 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.041741 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.041747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:32.043734 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:32.043754 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.045051 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.045060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:32.045287 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:32.049375 (+ 4088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:32.049392 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:32.050160 (+ 768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:32.053783 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:32.053787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:32.053790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:32.054240 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:32.054629 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:32.054632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:32.054634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:32.054820 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:32.055401 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:32.055407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:32.055593 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 23:21:32.055739 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 23:21:32.056049 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":368,"threads_started":1}
I20260620 23:21:32.057929 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 1582
I20260620 23:21:32.058127 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:21:32.058386 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50793
I20260620 23:21:32.058868 388 external_mini_cluster.cc:1428] Setting key a0ae41c40e79db65da4ffc898ff17fe1
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:21:32.093940 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35187
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:32.418864 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:32.419114 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:32.419305 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:32.425982 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:32.426077 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:32.426218 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:21:32.426324 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:21:32.435842 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:32.437004 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:32.438225 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:32.450628 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:32.450830 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:32.450910 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:32.451804 1722 server_base.cc:1034] running on GCE node
I20260620 23:21:32.452396 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:32.453236 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:32.454465 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781997692454402 us; error 43 us; skew 500 ppm
I20260620 23:21:32.454774 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:32 dist-test-slave-5kzz krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:32.461792 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:21:32.463225 1722 webserver.cc:460] Webserver started at http://127.0.97.2:34995/ using document root <none> and password file <none>
I20260620 23:21:32.463876 1722 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:32.463999 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:32.464314 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:32.466989 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "fe84d0ec6631460bbb104ee8815ff754"
format_stamp: "Formatted at 2026-06-20 23:21:32 on dist-test-slave-5kzz"
server_key: "e03b1bdac416dd1dcfde9eecb05d411b"
server_key_iv: "9219a4b9b56f9aa6c01b6a00c2b4fdc6"
server_key_version: "encryptionkey@0"
I20260620 23:21:32.467741 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fe84d0ec6631460bbb104ee8815ff754"
format_stamp: "Formatted at 2026-06-20 23:21:32 on dist-test-slave-5kzz"
server_key: "e03b1bdac416dd1dcfde9eecb05d411b"
server_key_iv: "9219a4b9b56f9aa6c01b6a00c2b4fdc6"
server_key_version: "encryptionkey@0"
I20260620 23:21:32.472512 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:21:32.475716 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:32.476940 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:21:32.477106 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "fe84d0ec6631460bbb104ee8815ff754"
format_stamp: "Formatted at 2026-06-20 23:21:32 on dist-test-slave-5kzz"
server_key: "e03b1bdac416dd1dcfde9eecb05d411b"
server_key_iv: "9219a4b9b56f9aa6c01b6a00c2b4fdc6"
server_key_version: "encryptionkey@0"
I20260620 23:21:32.477315 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:32.497100 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:32.500803 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:32.501057 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:32.503749 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:32.503858 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:32.503990 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:32.504060 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:21:32 dist-test-slave-5kzz krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:32.526518 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:32.504088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54249 (local address 127.0.97.62:35187)
0620 23:21:32.504244 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:32.504270 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:32.505730 (+ 1460us) server_negotiation.cc:407] Connection header received
0620 23:21:32.507873 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:32.507878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:32.507970 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:32.508078 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:32.510147 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.511185 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.514230 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.514855 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.514999 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:32.519737 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:32.519742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:32.519745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:32.519774 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:32.523166 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:32.524052 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:32.524058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:32.524063 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:32.524212 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:32.524845 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:32.524852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:32.524857 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:32.525145 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:32.525291 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:32.525956 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 23:21:32.526164 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:21:32.527500 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:32.504738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35187 (local address 127.0.97.2:54249)
0620 23:21:32.505582 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:32.505615 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:32.507577 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:32.508349 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:32.508364 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:32.509099 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:32.509980 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:32.509999 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.511362 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.511373 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:32.514068 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:32.514084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.515111 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.515123 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:32.515464 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:32.516201 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:32.516244 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:32.519496 (+ 3252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:32.523303 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:32.523314 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:32.523329 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:32.523929 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:32.524371 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:32.524376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:32.524382 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:32.524678 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:32.525296 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:32.525306 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:32.525618 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 23:21:32.525880 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 23:21:32.526362 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":446,"threads_started":1}
I20260620 23:21:32.560451 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43393
I20260620 23:21:32.560496 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43393 every 8 connection(s)
I20260620 23:21:32.561761 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:21:32.564185 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35187
I20260620 23:21:32.564414 1857 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:32.565196 1857 heartbeater.cc:511] Master 127.0.97.62:35187 requested a full tablet report, sending...
I20260620 23:21:32.566620 1511 ts_manager.cc:194] Registered new tserver with Master: fe84d0ec6631460bbb104ee8815ff754 (127.0.97.2:43393)
I20260620 23:21:32.566715 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:32.545280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44989 (local address 127.0.97.62:35187)
0620 23:21:32.545512 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:32.545522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:32.552362 (+ 6840us) server_negotiation.cc:407] Connection header received
0620 23:21:32.552623 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:32.552630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:32.552720 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:32.552809 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:32.554545 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.555657 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.557664 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.558066 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.558203 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:32.560509 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:32.560515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:32.560522 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:32.560561 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:32.563888 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:32.564704 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:32.564708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:32.564711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:32.564829 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:32.565199 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:32.565202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:32.565204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:32.565557 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:32.565682 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:32.566126 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 23:21:32.566332 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:21:32.566784 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:32.545386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35187 (local address 127.0.97.2:44989)
0620 23:21:32.552231 (+ 6845us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:32.552252 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:32.552430 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:32.553079 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:32.553089 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:32.553808 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:32.554289 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:32.554307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.555791 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.555797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:32.557533 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:32.557544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.558277 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.558284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:32.558532 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:32.559305 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:32.559325 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:32.560316 (+ 991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:32.564030 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:32.564035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:32.564040 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:32.564594 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:32.564942 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:32.564946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:32.564949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:32.565096 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:32.565691 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:32.565695 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:32.565891 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 23:21:32.566039 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 23:21:32.566327 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6675,"thread_start_us":6844,"threads_started":1}
I20260620 23:21:32.567548 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54249
I20260620 23:21:32.569680 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 1722
I20260620 23:21:32.569912 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:21:32.570461 388 external_mini_cluster.cc:1428] Setting key ca1131f0ee3cf737e5f4b4c69a776b31
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:21:32.605394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35187
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:32.909695 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:32.909953 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:32.910044 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:32.917071 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:32.917174 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:32.917255 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:21:32.917333 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:21:32.926553 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:32.927805 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:32.929116 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:32.942173 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:32.942231 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:32.942298 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:32.942920 1861 server_base.cc:1034] running on GCE node
I20260620 23:21:32.943436 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:32.944217 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:32.945420 1861 hybrid_clock.cc:648] HybridClock initialized: now 1781997692945363 us; error 47 us; skew 500 ppm
I20260620 23:21:32.945660 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:32 dist-test-slave-5kzz krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:32.948619 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:21:32.949855 1861 webserver.cc:460] Webserver started at http://127.0.97.3:34985/ using document root <none> and password file <none>
I20260620 23:21:32.950505 1861 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:32.950610 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:32.950959 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:32.953909 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "95cebdd59db84ab38a4450d3d33f9cc1"
format_stamp: "Formatted at 2026-06-20 23:21:32 on dist-test-slave-5kzz"
server_key: "8705caf6c156ac239693f404925d59b1"
server_key_iv: "1f766ad9e88c3261bf3aeaf4c8082b4a"
server_key_version: "encryptionkey@0"
I20260620 23:21:32.954737 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "95cebdd59db84ab38a4450d3d33f9cc1"
format_stamp: "Formatted at 2026-06-20 23:21:32 on dist-test-slave-5kzz"
server_key: "8705caf6c156ac239693f404925d59b1"
server_key_iv: "1f766ad9e88c3261bf3aeaf4c8082b4a"
server_key_version: "encryptionkey@0"
I20260620 23:21:32.959815 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:21:32.963373 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:32.964643 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:21:32.964797 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "95cebdd59db84ab38a4450d3d33f9cc1"
format_stamp: "Formatted at 2026-06-20 23:21:32 on dist-test-slave-5kzz"
server_key: "8705caf6c156ac239693f404925d59b1"
server_key_iv: "1f766ad9e88c3261bf3aeaf4c8082b4a"
server_key_version: "encryptionkey@0"
I20260620 23:21:32.965020 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:32.979802 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:32.983913 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:32.984184 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:32.986728 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:32.986857 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:32.987041 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:32.987121 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:21:32 dist-test-slave-5kzz krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:33.009516 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:32.988116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35187 (local address 127.0.97.3:43065)
0620 23:21:32.988934 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:32.988981 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:32.991090 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:32.991802 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:32.991812 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:32.992307 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:32.992848 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:32.992863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.994490 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.994498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:32.996675 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:32.996694 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.997731 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.997740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:32.997916 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:32.998609 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:32.998629 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:33.001618 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:33.005140 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:33.005149 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:33.005164 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:33.005579 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:33.005959 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:33.005965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:33.005969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:33.006151 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:33.006664 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:33.006672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:33.007093 (+ 421us) client_negotiation.cc:769] Sending connection context
0620 23:21:33.007449 (+ 356us) client_negotiation.cc:240] Negotiation successful
0620 23:21:33.008031 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":365,"threads_started":1}
I20260620 23:21:33.010535 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:32.987520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43065 (local address 127.0.97.62:35187)
0620 23:21:32.987733 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:32.987737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:32.989111 (+ 1374us) server_negotiation.cc:407] Connection header received
0620 23:21:32.991332 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:32.991340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:32.991449 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:32.991563 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:32.993026 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.994325 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.996863 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:32.997440 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:32.997592 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.001851 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:33.001856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:33.001859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:33.001896 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:33.005012 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:33.005705 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:33.005710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:33.005715 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:33.005844 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:33.006270 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:33.006277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:33.006286 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:33.006541 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:33.006690 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:33.010109 (+ 3419us) server_negotiation.cc:299] Negotiation successful
0620 23:21:33.010294 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:21:33.030920 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.016181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42371 (local address 127.0.97.62:35187)
0620 23:21:33.016436 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:33.016442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:33.016917 (+ 475us) server_negotiation.cc:407] Connection header received
0620 23:21:33.017112 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:33.017115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:33.017171 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:33.017250 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:33.018617 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.019550 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.021489 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.021914 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.022041 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.024346 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:33.024352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:33.024358 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:33.024401 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:33.028265 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:33.028876 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:33.028883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:33.028889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:33.029014 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:33.029381 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:33.029386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:33.029391 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:33.029727 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:33.029885 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:33.030326 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 23:21:33.030552 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:21:33.031067 1937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.016068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35187 (local address 127.0.97.3:42371)
0620 23:21:33.016774 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:33.016797 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:33.016981 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:33.017315 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:33.017323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:33.018024 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:33.018482 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.018495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.019779 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.019788 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.021311 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.021323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.022102 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.022109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.022278 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.023027 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:33.023056 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:33.024140 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:33.028387 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:33.028394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:33.028401 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:33.028764 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:33.029123 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:33.029129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:33.029132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:33.029247 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:33.029842 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:33.029846 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:33.030050 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 23:21:33.030218 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 23:21:33.030556 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":347,"threads_started":1}
I20260620 23:21:33.041785 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36179
I20260620 23:21:33.041803 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36179 every 8 connection(s)
I20260620 23:21:33.043085 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:21:33.045130 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35187
I20260620 23:21:33.045351 1996 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:33.046113 1996 heartbeater.cc:511] Master 127.0.97.62:35187 requested a full tablet report, sending...
I20260620 23:21:33.047461 1511 ts_manager.cc:194] Registered new tserver with Master: 95cebdd59db84ab38a4450d3d33f9cc1 (127.0.97.3:36179)
I20260620 23:21:33.048288 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43065
I20260620 23:21:33.050452 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 1861
I20260620 23:21:33.050638 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:21:33.051083 388 external_mini_cluster.cc:1428] Setting key ad2fe0dceb7c8609bcb9de2eb877739b
I20260620 23:21:33.053949 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:21:33.061416 1718 heartbeater.cc:503] Master 127.0.97.62:35187 was elected leader, sending a full tablet report...
Jun 20 23:21:33 dist-test-slave-5kzz krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997693, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:21:33 dist-test-slave-5kzz krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997693, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:33.098083 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.084983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55412 (local address 127.0.97.62:35187)
0620 23:21:33.085160 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:33.085165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:33.085453 (+ 288us) server_negotiation.cc:407] Connection header received
0620 23:21:33.085594 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:33.085598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:33.085661 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:33.085754 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:33.087316 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.088047 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.089239 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.089786 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.089898 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.093550 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:33.093554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:33.093558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:33.093588 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:33.096108 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:33.096625 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:33.096629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:33.096631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:33.096733 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:33.097026 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:33.097029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:33.097030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:33.097262 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:33.097443 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:33.097664 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 23:21:33.097817 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:21:33.102170 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:21:33.108731 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:21:33.136436 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.120009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43393 (local address 127.0.0.1:56410)
0620 23:21:33.121622 (+ 1613us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:33.121660 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:33.121813 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:33.124373 (+ 2560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:33.124398 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:33.124441 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:33.124949 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.124966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.128883 (+ 3917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.128891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.131996 (+ 3105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.132016 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.135774 (+ 3758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.135780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.135953 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.135991 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:21:33.136069 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 23:21:33.136191 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1377,"thread_start_us":2043,"threads_started":1}
I20260620 23:21:33.137291 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.120180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56410 (local address 127.0.97.2:43393)
0620 23:21:33.122475 (+ 2295us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:33.122485 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:33.122511 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:21:33.122602 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:33.122609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:33.122892 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:33.125220 (+ 2328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:33.125479 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.128698 (+ 3219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.133522 (+ 4824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.135543 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.136696 (+ 1153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.136878 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:33.136931 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 23:21:33.137045 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":701,"threads_started":1}
I20260620 23:21:33.141036 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 31e0b9f709ae45a78f8a360c6e63dc94 (DEFAULT_TABLE table=test-table [id=6f7ed8da382c4cedafd571a097b1821c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:33.142431 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e0b9f709ae45a78f8a360c6e63dc94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:33.144996 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.122531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41781 (local address 127.0.0.1:33952)
0620 23:21:33.135090 (+ 12559us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:33.135113 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:33.135269 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:33.138280 (+ 3011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:33.138287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:33.138320 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:33.138789 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.138804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.140252 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.140257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.142212 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.142231 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.144269 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.144275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.144441 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.144477 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:21:33.144567 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:21:33.144688 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12407,"thread_start_us":12827,"threads_started":1}
I20260620 23:21:33.145282 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.122791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.97.1:41781)
0620 23:21:33.137345 (+ 14554us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:33.137356 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:33.137389 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:21:33.137502 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:33.137513 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:33.137809 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:33.138033 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:33.138996 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.140119 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.142391 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.143828 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.143951 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.144138 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:33.144635 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 23:21:33.144796 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12986,"thread_start_us":12833,"threads_started":1}
I20260620 23:21:33.145712 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.120614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36179 (local address 127.0.0.1:42042)
0620 23:21:33.122246 (+ 1632us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:33.122259 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:33.122331 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:33.139786 (+ 17455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:33.139791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:33.139816 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:33.140121 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.140131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.141461 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.141485 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.143338 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.143353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.145137 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.145146 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.145268 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.145288 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 23:21:33.145366 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 23:21:33.145492 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1544,"thread_start_us":650,"threads_started":1}
I20260620 23:21:33.146852 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.121145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.0.97.3:36179)
0620 23:21:33.139254 (+ 18109us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:33.139262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:33.139293 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:21:33.139374 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:33.139381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:33.139584 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:33.139767 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:33.140334 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.141310 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.143520 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.144924 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.146096 (+ 1172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.146315 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:33.146379 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 23:21:33.146513 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17952,"thread_start_us":17781,"threads_started":1}
I20260620 23:21:33.149350 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 31e0b9f709ae45a78f8a360c6e63dc94 (DEFAULT_TABLE table=test-table [id=6f7ed8da382c4cedafd571a097b1821c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:33.149322 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 31e0b9f709ae45a78f8a360c6e63dc94 (DEFAULT_TABLE table=test-table [id=6f7ed8da382c4cedafd571a097b1821c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:33.150629 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e0b9f709ae45a78f8a360c6e63dc94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:33.150664 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e0b9f709ae45a78f8a360c6e63dc94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:33.155901 2015 tablet_bootstrap.cc:492] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754: Bootstrap starting.
I20260620 23:21:33.159391 2015 tablet_bootstrap.cc:654] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:33.160683 2015 log.cc:824] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:33.162985 2017 tablet_bootstrap.cc:492] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1: Bootstrap starting.
I20260620 23:21:33.163677 2015 tablet_bootstrap.cc:492] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754: No bootstrap required, opened a new log
I20260620 23:21:33.163949 2015 ts_tablet_manager.cc:1397] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260620 23:21:33.167361 2017 tablet_bootstrap.cc:654] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:33.167318 2018 tablet_bootstrap.cc:492] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362: Bootstrap starting.
I20260620 23:21:33.168607 2017 log.cc:824] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:33.171238 2018 tablet_bootstrap.cc:654] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:33.171823 2017 tablet_bootstrap.cc:492] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1: No bootstrap required, opened a new log
I20260620 23:21:33.172139 2017 ts_tablet_manager.cc:1397] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 23:21:33.172443 2018 log.cc:824] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:33.173247 2015 raft_consensus.cc:348] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.173599 2015 raft_consensus.cc:374] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:33.173697 2015 raft_consensus.cc:729] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe84d0ec6631460bbb104ee8815ff754, State: Initialized, Role: FOLLOWER
I20260620 23:21:33.174304 2015 consensus_queue.cc:260] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.175421 2018 tablet_bootstrap.cc:492] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362: No bootstrap required, opened a new log
I20260620 23:21:33.175679 2018 ts_tablet_manager.cc:1397] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260620 23:21:33.175846 1857 heartbeater.cc:503] Master 127.0.97.62:35187 was elected leader, sending a full tablet report...
I20260620 23:21:33.177321 2015 ts_tablet_manager.cc:1428] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260620 23:21:33.181483 2017 raft_consensus.cc:348] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.181898 2017 raft_consensus.cc:374] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:33.182003 2017 raft_consensus.cc:729] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95cebdd59db84ab38a4450d3d33f9cc1, State: Initialized, Role: FOLLOWER
I20260620 23:21:33.182888 2017 consensus_queue.cc:260] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.184270 2017 ts_tablet_manager.cc:1428] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 23:21:33.184368 1996 heartbeater.cc:503] Master 127.0.97.62:35187 was elected leader, sending a full tablet report...
I20260620 23:21:33.184507 2018 raft_consensus.cc:348] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.184852 2018 raft_consensus.cc:374] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:33.184963 2018 raft_consensus.cc:729] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619207749c5143e9add13c4f3551c362, State: Initialized, Role: FOLLOWER
I20260620 23:21:33.185526 2018 consensus_queue.cc:260] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.186990 2018 ts_tablet_manager.cc:1428] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
W20260620 23:21:33.294727 1997 tablet.cc:2369] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:21:33.303658 1719 tablet.cc:2369] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:21:33.314180 1858 tablet.cc:2369] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:21:33.385905 2021 raft_consensus.cc:482] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:21:33.386224 2021 raft_consensus.cc:504] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.387862 2021 leader_election.cc:288] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 619207749c5143e9add13c4f3551c362 (127.0.97.1:41781), 95cebdd59db84ab38a4450d3d33f9cc1 (127.0.97.3:36179)
I20260620 23:21:33.395475 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.388257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52253 (local address 127.0.97.3:36179)
0620 23:21:33.388468 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:33.388474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:33.389246 (+ 772us) server_negotiation.cc:407] Connection header received
0620 23:21:33.389431 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:33.389437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:33.389513 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:33.389605 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:33.390386 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.391475 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.393757 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.394510 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.394659 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.394789 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:33.395089 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 23:21:33.395220 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:21:33.395474 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.388333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44633 (local address 127.0.97.1:41781)
0620 23:21:33.388503 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:33.388510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:33.389243 (+ 733us) server_negotiation.cc:407] Connection header received
0620 23:21:33.389431 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:33.389437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:33.389555 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:33.389640 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:33.390386 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.391469 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.393710 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.394475 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.394627 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.394751 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:33.395089 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 23:21:33.395211 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:21:33.395675 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.388288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41781 (local address 127.0.97.2:44633)
0620 23:21:33.389142 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:33.389158 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:33.389272 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:33.389654 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:33.389658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:33.389694 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:33.390225 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.390232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.391666 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.391669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.393555 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.393570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.394726 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.394733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.394924 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.394952 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:21:33.395064 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 23:21:33.395210 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":67,"spinlock_wait_cycles":5504,"thread_start_us":426,"threads_started":1}
I20260620 23:21:33.395695 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.388241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36179 (local address 127.0.97.2:52253)
0620 23:21:33.389091 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:33.389112 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:33.389304 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:33.389651 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:33.389656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:33.389688 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:33.390219 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.390231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.391641 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.391644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.393604 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.393617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.394744 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.394750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.394935 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.394956 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:21:33.395066 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:21:33.395235 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":129,"spinlock_wait_cycles":39040,"thread_start_us":443,"threads_started":1}
I20260620 23:21:33.396744 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e0b9f709ae45a78f8a360c6e63dc94" candidate_uuid: "fe84d0ec6631460bbb104ee8815ff754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" is_pre_election: true
I20260620 23:21:33.396744 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e0b9f709ae45a78f8a360c6e63dc94" candidate_uuid: "fe84d0ec6631460bbb104ee8815ff754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619207749c5143e9add13c4f3551c362" is_pre_election: true
I20260620 23:21:33.397298 1668 raft_consensus.cc:2454] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe84d0ec6631460bbb104ee8815ff754 in term 0.
I20260620 23:21:33.397300 1951 raft_consensus.cc:2454] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe84d0ec6631460bbb104ee8815ff754 in term 0.
I20260620 23:21:33.397972 1740 leader_election.cc:302] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95cebdd59db84ab38a4450d3d33f9cc1, fe84d0ec6631460bbb104ee8815ff754; no voters:
I20260620 23:21:33.398281 2021 raft_consensus.cc:2781] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:21:33.398444 2021 raft_consensus.cc:482] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:21:33.398554 2021 raft_consensus.cc:3037] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:33.400724 2021 raft_consensus.cc:504] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.401342 2021 leader_election.cc:288] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [CANDIDATE]: Term 1 election: Requested vote from peers 619207749c5143e9add13c4f3551c362 (127.0.97.1:41781), 95cebdd59db84ab38a4450d3d33f9cc1 (127.0.97.3:36179)
I20260620 23:21:33.401713 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e0b9f709ae45a78f8a360c6e63dc94" candidate_uuid: "fe84d0ec6631460bbb104ee8815ff754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619207749c5143e9add13c4f3551c362"
I20260620 23:21:33.401783 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e0b9f709ae45a78f8a360c6e63dc94" candidate_uuid: "fe84d0ec6631460bbb104ee8815ff754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95cebdd59db84ab38a4450d3d33f9cc1"
I20260620 23:21:33.401917 1668 raft_consensus.cc:3037] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:33.401997 1951 raft_consensus.cc:3037] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:33.404134 1668 raft_consensus.cc:2454] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe84d0ec6631460bbb104ee8815ff754 in term 1.
I20260620 23:21:33.404161 1951 raft_consensus.cc:2454] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe84d0ec6631460bbb104ee8815ff754 in term 1.
I20260620 23:21:33.404544 1739 leader_election.cc:302] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619207749c5143e9add13c4f3551c362, fe84d0ec6631460bbb104ee8815ff754; no voters:
I20260620 23:21:33.404838 2021 raft_consensus.cc:2781] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:33.405715 2021 raft_consensus.cc:686] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 1 LEADER]: Becoming Leader. State: Replica: fe84d0ec6631460bbb104ee8815ff754, State: Running, Role: LEADER
I20260620 23:21:33.406162 2021 consensus_queue.cc:237] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.410669 1510 catalog_manager.cc:5495] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe84d0ec6631460bbb104ee8815ff754 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "fe84d0ec6631460bbb104ee8815ff754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } health_report { overall_health: UNKNOWN } } }
I20260620 23:21:33.500272 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.495029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.0.97.2:43393)
0620 23:21:33.495622 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:33.495628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:33.495649 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:21:33.495850 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:33.495854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:33.495924 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:33.496034 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:21:33.496333 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.497036 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.498288 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.498667 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.498814 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.499224 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:21:33.499698 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:21:33.499801 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:33.499902 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 23:21:33.499960 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:21:33.507855 1951 raft_consensus.cc:1261] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 1 FOLLOWER]: Refusing update from remote peer fe84d0ec6631460bbb104ee8815ff754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:33.507893 1668 raft_consensus.cc:1261] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362 [term 1 FOLLOWER]: Refusing update from remote peer fe84d0ec6631460bbb104ee8815ff754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:33.508726 2026 consensus_queue.cc:1035] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:33.509027 2021 consensus_queue.cc:1035] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:33.520576 2029 mvcc.cc:205] Tried to move back new op lower bound from 7299062552599592960 to 7299062552195256320. Current Snapshot: MvccSnapshot[applied={T|T < 7299062552599592960}]
I20260620 23:21:33.539188 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:33.539434 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:33.542904 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd: Sending DeleteTablet for 3 replicas of tablet 31e0b9f709ae45a78f8a360c6e63dc94
I20260620 23:21:33.543926 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e0b9f709ae45a78f8a360c6e63dc94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56410
I20260620 23:21:33.544047 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e0b9f709ae45a78f8a360c6e63dc94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33952
I20260620 23:21:33.544214 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e0b9f709ae45a78f8a360c6e63dc94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42042
I20260620 23:21:33.546452 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:21:33.547333 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:21:33.557420 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 85c006549cb64dddaf097231abd30965 (DEFAULT_TABLE table=test-table [id=12456e47ad01401c9e63574517af5377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:33.558014 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c006549cb64dddaf097231abd30965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:33.563812 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 85c006549cb64dddaf097231abd30965 (DEFAULT_TABLE table=test-table [id=12456e47ad01401c9e63574517af5377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:33.564025 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 85c006549cb64dddaf097231abd30965 (DEFAULT_TABLE table=test-table [id=12456e47ad01401c9e63574517af5377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:33.564358 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c006549cb64dddaf097231abd30965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:33.564630 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c006549cb64dddaf097231abd30965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:33.572731 2015 tablet_bootstrap.cc:492] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754: Bootstrap starting.
I20260620 23:21:33.575063 2015 tablet_bootstrap.cc:654] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:33.583338 2017 tablet_bootstrap.cc:492] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1: Bootstrap starting.
I20260620 23:21:33.584983 2017 tablet_bootstrap.cc:654] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:33.585503 2018 tablet_bootstrap.cc:492] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362: Bootstrap starting.
I20260620 23:21:33.587778 2018 tablet_bootstrap.cc:654] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:33.593410 2041 tablet_replica.cc:330] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1: stopping tablet replica
I20260620 23:21:33.593977 2041 raft_consensus.cc:2229] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:33.595398 2017 tablet_bootstrap.cc:492] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1: No bootstrap required, opened a new log
I20260620 23:21:33.595553 2017 ts_tablet_manager.cc:1397] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260620 23:21:33.595896 2039 tablet_replica.cc:330] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754: stopping tablet replica
I20260620 23:21:33.596335 2039 raft_consensus.cc:2229] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:21:33.596349 2017 raft_consensus.cc:348] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.596576 2017 raft_consensus.cc:374] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:33.596674 2017 raft_consensus.cc:729] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95cebdd59db84ab38a4450d3d33f9cc1, State: Initialized, Role: FOLLOWER
I20260620 23:21:33.597020 2039 raft_consensus.cc:2258] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:33.596974 2017 consensus_queue.cc:260] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.598232 2040 tablet_replica.cc:330] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362: stopping tablet replica
I20260620 23:21:33.598644 2040 raft_consensus.cc:2229] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:33.599469 2015 tablet_bootstrap.cc:492] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754: No bootstrap required, opened a new log
I20260620 23:21:33.599612 2015 ts_tablet_manager.cc:1397] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.008s
I20260620 23:21:33.600301 2041 raft_consensus.cc:2258] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:33.600492 2017 ts_tablet_manager.cc:1428] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260620 23:21:33.600386 2015 raft_consensus.cc:348] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.600603 2015 raft_consensus.cc:374] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:33.600703 2015 raft_consensus.cc:729] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe84d0ec6631460bbb104ee8815ff754, State: Initialized, Role: FOLLOWER
I20260620 23:21:33.600996 2015 consensus_queue.cc:260] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.602551 2041 ts_tablet_manager.cc:1905] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:33.603508 2015 ts_tablet_manager.cc:1428] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260620 23:21:33.599799 2018 tablet_bootstrap.cc:492] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362: No bootstrap required, opened a new log
I20260620 23:21:33.606913 2018 ts_tablet_manager.cc:1397] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.010s
I20260620 23:21:33.607692 2018 raft_consensus.cc:348] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.607930 2018 raft_consensus.cc:374] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:33.608029 2018 raft_consensus.cc:729] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619207749c5143e9add13c4f3551c362, State: Initialized, Role: FOLLOWER
I20260620 23:21:33.608269 2018 consensus_queue.cc:260] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.608803 2018 ts_tablet_manager.cc:1428] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:21:33.609853 2041 ts_tablet_manager.cc:1918] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:21:33.609999 2041 log.cc:1196] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/31e0b9f709ae45a78f8a360c6e63dc94
I20260620 23:21:33.610491 2041 ts_tablet_manager.cc:1939] T 31e0b9f709ae45a78f8a360c6e63dc94 P 95cebdd59db84ab38a4450d3d33f9cc1: Deleting consensus metadata
I20260620 23:21:33.599189 2040 raft_consensus.cc:2258] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:33.612286 1497 catalog_manager.cc:4841] TS 95cebdd59db84ab38a4450d3d33f9cc1 (127.0.97.3:36179): tablet 31e0b9f709ae45a78f8a360c6e63dc94 (table test-table [id=6f7ed8da382c4cedafd571a097b1821c]) successfully deleted
I20260620 23:21:33.613616 2039 ts_tablet_manager.cc:1905] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:33.614362 2040 ts_tablet_manager.cc:1905] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:33.619704 2039 ts_tablet_manager.cc:1918] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:21:33.619894 2039 log.cc:1196] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/31e0b9f709ae45a78f8a360c6e63dc94
I20260620 23:21:33.620723 2039 ts_tablet_manager.cc:1939] T 31e0b9f709ae45a78f8a360c6e63dc94 P fe84d0ec6631460bbb104ee8815ff754: Deleting consensus metadata
I20260620 23:21:33.620906 2040 ts_tablet_manager.cc:1918] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:21:33.621032 2040 log.cc:1196] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/31e0b9f709ae45a78f8a360c6e63dc94
I20260620 23:21:33.621487 2040 ts_tablet_manager.cc:1939] T 31e0b9f709ae45a78f8a360c6e63dc94 P 619207749c5143e9add13c4f3551c362: Deleting consensus metadata
I20260620 23:21:33.622434 1496 catalog_manager.cc:4841] TS fe84d0ec6631460bbb104ee8815ff754 (127.0.97.2:43393): tablet 31e0b9f709ae45a78f8a360c6e63dc94 (table test-table [id=6f7ed8da382c4cedafd571a097b1821c]) successfully deleted
I20260620 23:21:33.623025 1496 catalog_manager.cc:4841] TS 619207749c5143e9add13c4f3551c362 (127.0.97.1:41781): tablet 31e0b9f709ae45a78f8a360c6e63dc94 (table test-table [id=6f7ed8da382c4cedafd571a097b1821c]) successfully deleted
I20260620 23:21:33.770891 2034 raft_consensus.cc:482] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:21:33.771121 2034 raft_consensus.cc:504] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.772032 2034 leader_election.cc:288] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 619207749c5143e9add13c4f3551c362 (127.0.97.1:41781), 95cebdd59db84ab38a4450d3d33f9cc1 (127.0.97.3:36179)
I20260620 23:21:33.772464 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c006549cb64dddaf097231abd30965" candidate_uuid: "fe84d0ec6631460bbb104ee8815ff754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619207749c5143e9add13c4f3551c362" is_pre_election: true
I20260620 23:21:33.772716 1668 raft_consensus.cc:2454] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe84d0ec6631460bbb104ee8815ff754 in term 0.
I20260620 23:21:33.772665 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c006549cb64dddaf097231abd30965" candidate_uuid: "fe84d0ec6631460bbb104ee8815ff754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" is_pre_election: true
I20260620 23:21:33.772910 1951 raft_consensus.cc:2454] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe84d0ec6631460bbb104ee8815ff754 in term 0.
I20260620 23:21:33.773138 1739 leader_election.cc:302] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619207749c5143e9add13c4f3551c362, fe84d0ec6631460bbb104ee8815ff754; no voters:
I20260620 23:21:33.773396 2034 raft_consensus.cc:2781] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:21:33.773516 2034 raft_consensus.cc:482] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:21:33.773608 2034 raft_consensus.cc:3037] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:33.775864 2034 raft_consensus.cc:504] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.776464 2034 leader_election.cc:288] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [CANDIDATE]: Term 1 election: Requested vote from peers 619207749c5143e9add13c4f3551c362 (127.0.97.1:41781), 95cebdd59db84ab38a4450d3d33f9cc1 (127.0.97.3:36179)
I20260620 23:21:33.776891 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c006549cb64dddaf097231abd30965" candidate_uuid: "fe84d0ec6631460bbb104ee8815ff754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619207749c5143e9add13c4f3551c362"
I20260620 23:21:33.776947 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c006549cb64dddaf097231abd30965" candidate_uuid: "fe84d0ec6631460bbb104ee8815ff754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95cebdd59db84ab38a4450d3d33f9cc1"
I20260620 23:21:33.777070 1668 raft_consensus.cc:3037] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:33.777585 1951 raft_consensus.cc:3037] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:33.779222 1668 raft_consensus.cc:2454] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe84d0ec6631460bbb104ee8815ff754 in term 1.
I20260620 23:21:33.779629 1739 leader_election.cc:302] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619207749c5143e9add13c4f3551c362, fe84d0ec6631460bbb104ee8815ff754; no voters:
I20260620 23:21:33.779892 2034 raft_consensus.cc:2781] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:33.780055 2034 raft_consensus.cc:686] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 1 LEADER]: Becoming Leader. State: Replica: fe84d0ec6631460bbb104ee8815ff754, State: Running, Role: LEADER
I20260620 23:21:33.780440 1951 raft_consensus.cc:2454] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe84d0ec6631460bbb104ee8815ff754 in term 1.
I20260620 23:21:33.780346 2034 consensus_queue.cc:237] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.783231 1511 catalog_manager.cc:5495] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe84d0ec6631460bbb104ee8815ff754 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "fe84d0ec6631460bbb104ee8815ff754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:21:33 dist-test-slave-5kzz krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:33.858824 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.844922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35187 (local address 127.0.0.1:55420)
0620 23:21:33.845917 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:33.845932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:33.846064 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:33.846479 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:33.846482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:33.847189 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:33.847480 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.847491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.848522 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.848528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.849445 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:33.849455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.850099 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.850104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:33.850332 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.851031 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:33.851051 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:33.853059 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:33.856214 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:33.856225 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:33.856233 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:33.856674 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:33.857019 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:33.857026 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:33.857030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:33.857180 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:33.857723 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:33.857730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:33.857896 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 23:21:33.858056 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 23:21:33.858321 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":41,"thread_start_us":727,"threads_started":1}
I20260620 23:21:33.858844 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.844966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.0.97.62:35187)
0620 23:21:33.845688 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:33.845699 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:33.846046 (+ 347us) server_negotiation.cc:407] Connection header received
0620 23:21:33.846186 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:33.846192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:33.846291 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:33.846463 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:33.847625 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.848354 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.849573 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:33.849955 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:33.850085 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:33.853235 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:33.853241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:33.853245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:33.853279 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:33.856071 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:33.856784 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:33.856790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:33.856793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:33.856918 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:33.857282 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:33.857289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:33.857293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:33.857593 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:33.857747 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:33.858134 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 23:21:33.858444 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":553,"thread_start_us":390,"threads_started":1}
I20260620 23:21:33.863725 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55420:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:21:33.864724 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:21:33.871749 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 3b736569b96242aeae9d6ac8e5ecea00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4546a71ff780430cb8078bf315f08d5d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:21:33.872330 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b736569b96242aeae9d6ac8e5ecea00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:33.872151 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 3b736569b96242aeae9d6ac8e5ecea00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4546a71ff780430cb8078bf315f08d5d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:21:33.872742 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b736569b96242aeae9d6ac8e5ecea00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:33.878468 2018 tablet_bootstrap.cc:492] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: Bootstrap starting.
I20260620 23:21:33.878731 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 3b736569b96242aeae9d6ac8e5ecea00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4546a71ff780430cb8078bf315f08d5d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:21:33.879297 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b736569b96242aeae9d6ac8e5ecea00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:33.879843 2018 tablet_bootstrap.cc:654] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:33.880033 2017 tablet_bootstrap.cc:492] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1: Bootstrap starting.
I20260620 23:21:33.881749 2017 tablet_bootstrap.cc:654] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:33.882803 2018 tablet_bootstrap.cc:492] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: No bootstrap required, opened a new log
I20260620 23:21:33.882963 2018 ts_tablet_manager.cc:1397] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 23:21:33.883843 2018 raft_consensus.cc:348] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.884164 2018 raft_consensus.cc:374] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:33.884255 2018 raft_consensus.cc:729] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619207749c5143e9add13c4f3551c362, State: Initialized, Role: FOLLOWER
I20260620 23:21:33.884660 2018 consensus_queue.cc:260] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.885462 2023 tablet_replica.cc:439] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } } }
I20260620 23:21:33.885897 2023 tablet_replica.cc:442] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:33.886044 2015 tablet_bootstrap.cc:492] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754: Bootstrap starting.
I20260620 23:21:33.887944 2015 tablet_bootstrap.cc:654] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:33.889488 2018 ts_tablet_manager.cc:1428] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 23:21:33.890084 2017 tablet_bootstrap.cc:492] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1: No bootstrap required, opened a new log
I20260620 23:21:33.890244 2017 ts_tablet_manager.cc:1397] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260620 23:21:33.891067 2017 raft_consensus.cc:348] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.891314 2017 raft_consensus.cc:374] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:33.891448 2017 raft_consensus.cc:729] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95cebdd59db84ab38a4450d3d33f9cc1, State: Initialized, Role: FOLLOWER
I20260620 23:21:33.891747 2017 consensus_queue.cc:260] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.892215 2022 tablet_replica.cc:439] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } } }
I20260620 23:21:33.892571 2022 tablet_replica.cc:442] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:33.893043 2017 ts_tablet_manager.cc:1428] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 23:21:33.893144 2015 tablet_bootstrap.cc:492] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754: No bootstrap required, opened a new log
I20260620 23:21:33.893339 2015 ts_tablet_manager.cc:1397] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 23:21:33.894594 2015 raft_consensus.cc:348] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.894881 2015 raft_consensus.cc:374] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:33.894977 2015 raft_consensus.cc:729] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe84d0ec6631460bbb104ee8815ff754, State: Initialized, Role: FOLLOWER
I20260620 23:21:33.895238 2015 consensus_queue.cc:260] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.895663 2034 tablet_replica.cc:439] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } } }
I20260620 23:21:33.895845 2015 ts_tablet_manager.cc:1428] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:21:33.895931 2034 tablet_replica.cc:442] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:33.993063 2023 raft_consensus.cc:482] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:21:33.993279 2023 raft_consensus.cc:504] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:33.995177 2023 leader_election.cc:288] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe84d0ec6631460bbb104ee8815ff754 (127.0.97.2:43393), 95cebdd59db84ab38a4450d3d33f9cc1 (127.0.97.3:36179)
I20260620 23:21:34.014840 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.995412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36179 (local address 127.0.97.1:49477)
0620 23:21:34.007762 (+ 12350us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:34.007790 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:34.007964 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:34.008699 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:34.008704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:34.008740 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:34.009051 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:34.009064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.010231 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.010239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:34.012366 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:34.012380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.013963 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.013970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:34.014131 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.014179 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 23:21:34.014269 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:21:34.014426 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12225,"mutex_wait_us":25,"spinlock_wait_cycles":2944,"thread_start_us":12841,"threads_started":1}
I20260620 23:21:34.014838 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.995713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49477 (local address 127.0.97.3:36179)
0620 23:21:34.008396 (+ 12683us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:34.008402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:34.008418 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:21:34.008484 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:34.008489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:34.008555 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:34.008650 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:34.009278 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.010072 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.012530 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.013809 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.013906 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.013973 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:34.014341 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 23:21:34.014473 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12539,"thread_start_us":12427,"threads_started":1}
I20260620 23:21:34.015163 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.996027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53079 (local address 127.0.97.2:43393)
0620 23:21:33.996201 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:33.996206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:34.008187 (+ 11981us) server_negotiation.cc:407] Connection header received
0620 23:21:34.008442 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:34.008448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:34.008518 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:34.008580 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:34.009389 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.010528 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.013264 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.014103 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.014241 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.014342 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:34.014740 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 23:21:34.014896 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:21:34.015432 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:33.995442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43393 (local address 127.0.97.1:53079)
0620 23:21:34.008090 (+ 12648us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:34.008109 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:34.008281 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:34.008785 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:34.008791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:34.008848 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:34.009233 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:34.009244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.010663 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.010670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:34.013111 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:34.013122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.014360 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.014365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:34.014523 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.014556 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:21:34.014742 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 23:21:34.014883 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12543,"mutex_wait_us":43,"spinlock_wait_cycles":3840,"thread_start_us":12102,"threads_started":1}
I20260620 23:21:34.015679 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b736569b96242aeae9d6ac8e5ecea00" candidate_uuid: "619207749c5143e9add13c4f3551c362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" is_pre_election: true
I20260620 23:21:34.015965 1951 raft_consensus.cc:2454] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619207749c5143e9add13c4f3551c362 in term 0.
I20260620 23:21:34.016398 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b736569b96242aeae9d6ac8e5ecea00" candidate_uuid: "619207749c5143e9add13c4f3551c362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe84d0ec6631460bbb104ee8815ff754" is_pre_election: true
I20260620 23:21:34.016582 1600 leader_election.cc:302] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619207749c5143e9add13c4f3551c362, 95cebdd59db84ab38a4450d3d33f9cc1; no voters:
I20260620 23:21:34.016702 1808 raft_consensus.cc:2454] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619207749c5143e9add13c4f3551c362 in term 0.
I20260620 23:21:34.016901 2023 raft_consensus.cc:2781] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:21:34.017009 2023 raft_consensus.cc:482] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:21:34.017114 2023 raft_consensus.cc:3037] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:34.019438 2023 raft_consensus.cc:504] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:34.019955 2023 leader_election.cc:288] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [CANDIDATE]: Term 1 election: Requested vote from peers fe84d0ec6631460bbb104ee8815ff754 (127.0.97.2:43393), 95cebdd59db84ab38a4450d3d33f9cc1 (127.0.97.3:36179)
I20260620 23:21:34.020287 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b736569b96242aeae9d6ac8e5ecea00" candidate_uuid: "619207749c5143e9add13c4f3551c362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe84d0ec6631460bbb104ee8815ff754"
I20260620 23:21:34.020454 1808 raft_consensus.cc:3037] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:34.020457 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b736569b96242aeae9d6ac8e5ecea00" candidate_uuid: "619207749c5143e9add13c4f3551c362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95cebdd59db84ab38a4450d3d33f9cc1"
I20260620 23:21:34.020680 1951 raft_consensus.cc:3037] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:34.022718 1808 raft_consensus.cc:2454] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619207749c5143e9add13c4f3551c362 in term 1.
I20260620 23:21:34.022974 1951 raft_consensus.cc:2454] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619207749c5143e9add13c4f3551c362 in term 1.
I20260620 23:21:34.023077 1599 leader_election.cc:302] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619207749c5143e9add13c4f3551c362, fe84d0ec6631460bbb104ee8815ff754; no voters:
I20260620 23:21:34.023393 2023 raft_consensus.cc:2781] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:34.023751 2023 raft_consensus.cc:686] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [term 1 LEADER]: Becoming Leader. State: Replica: 619207749c5143e9add13c4f3551c362, State: Running, Role: LEADER
I20260620 23:21:34.024116 2023 consensus_queue.cc:237] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } }
I20260620 23:21:34.025978 2059 tablet_replica.cc:439] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: TxnStatusTablet state changed. Reason: New leader 619207749c5143e9add13c4f3551c362. Latest consensus state: current_term: 1 leader_uuid: "619207749c5143e9add13c4f3551c362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } } }
I20260620 23:21:34.026337 2059 tablet_replica.cc:442] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:21:34.027510 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:21:34.027732 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:21:34.027798 1510 catalog_manager.cc:5495] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 reported cstate change: term changed from 0 to 1, leader changed from <none> to 619207749c5143e9add13c4f3551c362 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "619207749c5143e9add13c4f3551c362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } health_report { overall_health: UNKNOWN } } }
I20260620 23:21:34.080545 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:34.073753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41781 (local address 127.0.0.1:33958)
0620 23:21:34.074192 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:34.074218 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:34.074364 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:34.075054 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:34.075058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:34.075068 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:21:34.075211 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:34.075219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.076890 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.076894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:34.077803 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:34.077813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.078810 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.078814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:34.078954 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.079002 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:21:34.080084 (+ 1082us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:21:34.080096 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 23:21:34.080192 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:21:34.080293 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260620 23:21:34.081137 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:34.073738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.0.97.1:41781)
0620 23:21:34.074561 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:34.074568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:34.074592 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:21:34.074710 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:34.074715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:34.074816 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:34.075006 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:21:34.075374 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.076662 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.077952 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.078616 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.078780 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.079205 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:21:34.079953 (+ 748us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:21:34.080127 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:34.080228 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 23:21:34.080307 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":543,"threads_started":1}
I20260620 23:21:34.085299 1951 raft_consensus.cc:1261] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 1 FOLLOWER]: Refusing update from remote peer 619207749c5143e9add13c4f3551c362: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:34.085552 1808 raft_consensus.cc:1261] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754 [term 1 FOLLOWER]: Refusing update from remote peer 619207749c5143e9add13c4f3551c362: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:34.086062 2059 consensus_queue.cc:1035] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:34.086395 2023 consensus_queue.cc:1035] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:34.092248 2022 tablet_replica.cc:439] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1: TxnStatusTablet state changed. Reason: New leader 619207749c5143e9add13c4f3551c362. Latest consensus state: current_term: 1 leader_uuid: "619207749c5143e9add13c4f3551c362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } } }
I20260620 23:21:34.092540 2022 tablet_replica.cc:442] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:34.094915 2065 mvcc.cc:205] Tried to move back new op lower bound from 7299062554966855680 to 7299062554726223872. Current Snapshot: MvccSnapshot[applied={T|T < 7299062554966855680}]
I20260620 23:21:34.095505 2059 tablet_replica.cc:439] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "619207749c5143e9add13c4f3551c362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } } }
I20260620 23:21:34.095744 2059 tablet_replica.cc:442] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:21:34.096457 2022 tablet_replica.cc:439] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "619207749c5143e9add13c4f3551c362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } } }
I20260620 23:21:34.096673 2022 tablet_replica.cc:442] T 3b736569b96242aeae9d6ac8e5ecea00 P 95cebdd59db84ab38a4450d3d33f9cc1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:34.100246 2034 tablet_replica.cc:439] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754: TxnStatusTablet state changed. Reason: New leader 619207749c5143e9add13c4f3551c362. Latest consensus state: current_term: 1 leader_uuid: "619207749c5143e9add13c4f3551c362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } } }
I20260620 23:21:34.100548 2034 tablet_replica.cc:442] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:34.100947 2059 tablet_replica.cc:439] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "619207749c5143e9add13c4f3551c362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } } }
I20260620 23:21:34.101181 2059 tablet_replica.cc:442] T 3b736569b96242aeae9d6ac8e5ecea00 P 619207749c5143e9add13c4f3551c362: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:21:34.110344 2034 tablet_replica.cc:439] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "619207749c5143e9add13c4f3551c362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 } } peers { permanent_uuid: "fe84d0ec6631460bbb104ee8815ff754" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43393 } } peers { permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 } } }
I20260620 23:21:34.110597 2034 tablet_replica.cc:442] T 3b736569b96242aeae9d6ac8e5ecea00 P fe84d0ec6631460bbb104ee8815ff754: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:34.123840 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:34.116846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39369 (local address 127.0.97.1:41781)
0620 23:21:34.117049 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:34.117058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:34.119256 (+ 2198us) server_negotiation.cc:407] Connection header received
0620 23:21:34.119459 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:34.119463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:34.119519 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:34.119606 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:21:34.120049 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.120743 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.122017 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.122465 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.122607 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.123126 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:21:34.123260 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:21:34.123368 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:34.123464 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 23:21:34.123533 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:21:34.123921 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:34.116800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41781 (local address 127.0.97.2:39369)
0620 23:21:34.119143 (+ 2343us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:34.119157 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:34.119318 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:34.119631 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:34.119636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:34.119650 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:21:34.119914 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:34.119922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.120890 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.120896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:34.121884 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:34.121894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.122642 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.122649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:34.122803 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.122898 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:21:34.123361 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:21:34.123371 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:21:34.123449 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 23:21:34.123573 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2195,"thread_start_us":2070,"threads_started":1}
I20260620 23:21:34.132593 1951 raft_consensus.cc:1261] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 1 FOLLOWER]: Refusing update from remote peer fe84d0ec6631460bbb104ee8815ff754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:34.132840 1668 raft_consensus.cc:1261] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362 [term 1 FOLLOWER]: Refusing update from remote peer fe84d0ec6631460bbb104ee8815ff754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:34.133108 2034 consensus_queue.cc:1035] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95cebdd59db84ab38a4450d3d33f9cc1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:34.133445 2026 consensus_queue.cc:1035] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "619207749c5143e9add13c4f3551c362" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:34.140962 2066 mvcc.cc:205] Tried to move back new op lower bound from 7299062555162902528 to 7299062553726939136. Current Snapshot: MvccSnapshot[applied={T|T < 7299062555162902528}]
I20260620 23:21:34.171202 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:34.164125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50215 (local address 127.0.97.2:43393)
0620 23:21:34.164320 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:34.164327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:34.164933 (+ 606us) server_negotiation.cc:407] Connection header received
0620 23:21:34.165152 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:34.165157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:34.165237 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:34.165330 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:21:34.166500 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.167634 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.169340 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.169835 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.169985 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.170524 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:21:34.170660 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:21:34.170770 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:34.170870 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 23:21:34.170947 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:21:34.171245 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:34.164090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43393 (local address 127.0.97.1:50215)
0620 23:21:34.164828 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:34.164844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:34.165009 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:34.165342 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:34.165346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:34.165390 (+ 44us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:21:34.166259 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:34.166273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.167764 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.167768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:34.169181 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:34.169191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.170034 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.170043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:34.170196 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.170264 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:21:34.170771 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:21:34.170787 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 23:21:34.170852 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 23:21:34.170948 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":416,"threads_started":1}
I20260620 23:21:34.201876 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:34.202116 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:34.204885 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e74e4ab23a146a0933a2ea74891f0cd: Sending DeleteTablet for 3 replicas of tablet 85c006549cb64dddaf097231abd30965
I20260620 23:21:34.205556 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 85c006549cb64dddaf097231abd30965 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33952
I20260620 23:21:34.205786 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 85c006549cb64dddaf097231abd30965 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42042
I20260620 23:21:34.206005 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 85c006549cb64dddaf097231abd30965 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56410
I20260620 23:21:34.216156 2085 tablet_replica.cc:330] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754: stopping tablet replica
I20260620 23:21:34.216801 2085 raft_consensus.cc:2229] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:21:34.217373 2085 raft_consensus.cc:2258] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:34.218827 2085 ts_tablet_manager.cc:1905] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:34.219501 2083 tablet_replica.cc:330] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362: stopping tablet replica
I20260620 23:21:34.222954 2083 raft_consensus.cc:2229] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:34.223451 2083 raft_consensus.cc:2258] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:34.225308 2083 ts_tablet_manager.cc:1905] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:34.229773 2084 tablet_replica.cc:330] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1: stopping tablet replica
I20260620 23:21:34.229981 2084 raft_consensus.cc:2229] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:34.230240 2084 raft_consensus.cc:2258] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:34.231395 2084 ts_tablet_manager.cc:1905] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:34.235390 2085 ts_tablet_manager.cc:1918] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:21:34.235529 2085 log.cc:1196] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/85c006549cb64dddaf097231abd30965
I20260620 23:21:34.235978 2085 ts_tablet_manager.cc:1939] T 85c006549cb64dddaf097231abd30965 P fe84d0ec6631460bbb104ee8815ff754: Deleting consensus metadata
I20260620 23:21:34.237052 1496 catalog_manager.cc:4841] TS fe84d0ec6631460bbb104ee8815ff754 (127.0.97.2:43393): tablet 85c006549cb64dddaf097231abd30965 (table test-table [id=12456e47ad01401c9e63574517af5377]) successfully deleted
I20260620 23:21:34.239045 2084 ts_tablet_manager.cc:1918] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:21:34.239236 2084 log.cc:1196] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/85c006549cb64dddaf097231abd30965
I20260620 23:21:34.239740 2084 ts_tablet_manager.cc:1939] T 85c006549cb64dddaf097231abd30965 P 95cebdd59db84ab38a4450d3d33f9cc1: Deleting consensus metadata
Jun 20 23:21:34 dist-test-slave-5kzz krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997693, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:21:34.240833 1497 catalog_manager.cc:4841] TS 95cebdd59db84ab38a4450d3d33f9cc1 (127.0.97.3:36179): tablet 85c006549cb64dddaf097231abd30965 (table test-table [id=12456e47ad01401c9e63574517af5377]) successfully deleted
I20260620 23:21:34.241853 2083 ts_tablet_manager.cc:1918] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:21:34.242012 2083 log.cc:1196] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/85c006549cb64dddaf097231abd30965
I20260620 23:21:34.242513 2083 ts_tablet_manager.cc:1939] T 85c006549cb64dddaf097231abd30965 P 619207749c5143e9add13c4f3551c362: Deleting consensus metadata
I20260620 23:21:34.243774 1496 catalog_manager.cc:4841] TS 619207749c5143e9add13c4f3551c362 (127.0.97.1:41781): tablet 85c006549cb64dddaf097231abd30965 (table test-table [id=12456e47ad01401c9e63574517af5377]) successfully deleted
I20260620 23:21:34.249224 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:34.222844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.0.97.1:41781)
0620 23:21:34.223044 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:34.223050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:34.229942 (+ 6892us) server_negotiation.cc:407] Connection header received
0620 23:21:34.230136 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:34.230141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:34.230257 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:34.230383 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:34.232527 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.233746 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.235698 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.236349 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.236457 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.241637 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:34.241658 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:34.241677 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:34.241720 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:34.245306 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:34.246175 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:34.246183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:34.246185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:34.246865 (+ 680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:34.247283 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:34.247287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:34.247289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:34.247666 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:34.247902 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:34.248633 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 23:21:34.248819 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 23:21:34.250003 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33968
I20260620 23:21:34.266723 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:34.253505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.0.97.62:35187)
0620 23:21:34.253683 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:34.253689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:34.253898 (+ 209us) server_negotiation.cc:407] Connection header received
0620 23:21:34.254007 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:34.254012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:34.254090 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:34.254193 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:34.255424 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.256528 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.257712 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.258335 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.258474 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.260588 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:34.260594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:34.260599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:34.260633 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:34.264172 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:34.264843 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:34.264849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:34.264853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:34.264989 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:34.265310 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:34.265316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:34.265319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:34.265665 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:34.265900 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:34.266139 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 23:21:34.266353 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 23:21:34.267306 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55436
I20260620 23:21:34.269577 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 1582
I20260620 23:21:34.283659 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:34.270422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.0.97.1:41781)
0620 23:21:34.270618 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:34.270625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:34.270928 (+ 303us) server_negotiation.cc:407] Connection header received
0620 23:21:34.271030 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:34.271035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:34.271105 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:34.271217 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:34.272149 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.273229 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.274343 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.275113 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.275242 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.277273 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:34.277279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:34.277284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:34.277322 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:34.280684 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:34.281285 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:34.281291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:34.281294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:34.281437 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:34.282155 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:34.282160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:34.282164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:34.282525 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:34.282834 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:34.283050 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 23:21:34.283284 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 23:21:34.284241 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33982
W20260620 23:21:34.284806 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:21:34.284986 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 1582
I20260620 23:21:34.304149 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 1722
Jun 20 23:21:34 dist-test-slave-5kzz krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997693, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:21:34.317689 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:34.304775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.0.97.2:43393)
0620 23:21:34.304948 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:34.304955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:34.305028 (+ 73us) server_negotiation.cc:407] Connection header received
0620 23:21:34.305214 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:34.305219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:34.305296 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:34.305403 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:34.306430 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.307224 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.308446 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.308830 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.308955 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.312891 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:34.312902 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:34.312916 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:34.312945 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:34.315511 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:34.316069 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:34.316075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:34.316078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:34.316189 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:34.316471 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:34.316474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:34.316476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:34.316883 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:34.317091 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:34.317306 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 23:21:34.317462 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 23:21:34.318264 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56438
W20260620 23:21:34.318938 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:21:34.319103 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 1722
I20260620 23:21:34.339756 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 1861
Jun 20 23:21:34 dist-test-slave-5kzz krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997693, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:21:34.353633 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:34.340466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.0.97.3:36179)
0620 23:21:34.340637 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:34.340644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:34.340681 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:21:34.340897 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:34.340901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:34.340962 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:34.341066 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:34.342244 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.343004 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.344172 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:34.344579 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:34.344697 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:34.348617 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:34.348630 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:34.348646 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:34.348679 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:34.351458 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:34.352014 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:34.352021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:34.352023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:34.352148 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:34.352431 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:34.352434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:34.352436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:34.352719 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:34.352953 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:34.353135 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 23:21:34.353290 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 23:21:34.354243 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42054
W20260620 23:21:34.354990 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:21:34.355135 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 1861
I20260620 23:21:34.371282 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 1479
I20260620 23:21:34.372428 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:34.480885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 1479
2026-06-20T23:21:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4693 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:21:34 dist-test-slave-5kzz krb5kdc[2094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:21:34 dist-test-slave-5kzz krb5kdc[2094](info): set up 2 sockets
Jun 20 23:21:34 dist-test-slave-5kzz krb5kdc[2094](info): commencing operation
krb5kdc: starting...
W20260620 23:21:35.534863 388 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:21:35 dist-test-slave-5kzz krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:21:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:21:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:21:35.713615 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34167
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:36.020957 2107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:36.021212 2107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:36.021407 2107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:36.028385 2107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:21:36.028469 2107 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:21:36.028553 2107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:36.028662 2107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:36.028740 2107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:21:36.028824 2107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:21:36.038069 2107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34167
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:36.039347 2107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:36.040683 2107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:36.052091 2112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:36.052052 2115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:36.052145 2113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:36.053205 2107 server_base.cc:1034] running on GCE node
I20260620 23:21:36.053956 2107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:36.055357 2107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:36.056563 2107 hybrid_clock.cc:648] HybridClock initialized: now 1781997696056510 us; error 29 us; skew 500 ppm
I20260620 23:21:36.056908 2107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:36 dist-test-slave-5kzz krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:36.059962 2107 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:21:36.061326 2107 webserver.cc:460] Webserver started at http://127.0.97.62:41099/ using document root <none> and password file <none>
I20260620 23:21:36.061993 2107 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:36.062101 2107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:36.062417 2107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:36.065336 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "a69992dc0903466da1e7f57f7c747a8f"
format_stamp: "Formatted at 2026-06-20 23:21:36 on dist-test-slave-5kzz"
server_key: "417df548a5f598707d2b319f7d09a33c"
server_key_iv: "02791130b0ebb8feaf58835388e60236"
server_key_version: "encryptionkey@0"
I20260620 23:21:36.066079 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "a69992dc0903466da1e7f57f7c747a8f"
format_stamp: "Formatted at 2026-06-20 23:21:36 on dist-test-slave-5kzz"
server_key: "417df548a5f598707d2b319f7d09a33c"
server_key_iv: "02791130b0ebb8feaf58835388e60236"
server_key_version: "encryptionkey@0"
I20260620 23:21:36.071215 2107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 23:21:36.074453 2122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:36.075824 2107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:21:36.076032 2107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "a69992dc0903466da1e7f57f7c747a8f"
format_stamp: "Formatted at 2026-06-20 23:21:36 on dist-test-slave-5kzz"
server_key: "417df548a5f598707d2b319f7d09a33c"
server_key_iv: "02791130b0ebb8feaf58835388e60236"
server_key_version: "encryptionkey@0"
I20260620 23:21:36.076246 2107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:36.091707 2107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:36.095430 2107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:36.095687 2107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:36.116463 2107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34167
I20260620 23:21:36.116470 2183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34167 every 8 connection(s)
I20260620 23:21:36.117810 2107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:21:36.120257 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 2107
I20260620 23:21:36.120457 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:21:36.121253 388 external_mini_cluster.cc:1428] Setting key 6b57df628fdfb25a57011bb557238916
I20260620 23:21:36.122071 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:36.133781 2184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f: Bootstrap starting.
Jun 20 23:21:36 dist-test-slave-5kzz krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:36.137425 2184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:36.138581 2184 log.cc:824] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:36.141680 2184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f: No bootstrap required, opened a new log
I20260620 23:21:36.143872 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:36.123264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.0.97.62:34167)
0620 23:21:36.123970 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:36.123981 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:36.124021 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:21:36.126492 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:36.126512 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:36.126983 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:36.127370 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:36.128754 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:36.129988 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:36.131273 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:36.131997 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:36.132224 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:36.136507 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:36.136517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:36.136531 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:36.136564 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:36.139992 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:36.140748 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:36.140754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:36.140760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:36.140914 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:36.141286 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:36.141289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:36.141291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:36.141804 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:36.142088 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:36.142507 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 23:21:36.142935 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":320,"threads_started":1}
I20260620 23:21:36.149547 2184 raft_consensus.cc:348] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69992dc0903466da1e7f57f7c747a8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34167 } }
I20260620 23:21:36.149878 2184 raft_consensus.cc:374] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:36.149999 2184 raft_consensus.cc:729] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69992dc0903466da1e7f57f7c747a8f, State: Initialized, Role: FOLLOWER
I20260620 23:21:36.150514 2184 consensus_queue.cc:260] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69992dc0903466da1e7f57f7c747a8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34167 } }
I20260620 23:21:36.150760 2184 raft_consensus.cc:388] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:21:36.150878 2184 raft_consensus.cc:482] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:21:36.151003 2184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:36.153090 2184 raft_consensus.cc:504] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69992dc0903466da1e7f57f7c747a8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34167 } }
I20260620 23:21:36.153633 2184 leader_election.cc:302] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a69992dc0903466da1e7f57f7c747a8f; no voters:
I20260620 23:21:36.154218 2184 leader_election.cc:288] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:21:36.154348 2189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:36.155216 2189 raft_consensus.cc:686] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [term 1 LEADER]: Becoming Leader. State: Replica: a69992dc0903466da1e7f57f7c747a8f, State: Running, Role: LEADER
I20260620 23:21:36.155710 2189 consensus_queue.cc:237] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69992dc0903466da1e7f57f7c747a8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34167 } }
I20260620 23:21:36.156410 2184 sys_catalog.cc:564] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:21:36.158234 2191 sys_catalog.cc:455] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a69992dc0903466da1e7f57f7c747a8f. Latest consensus state: current_term: 1 leader_uuid: "a69992dc0903466da1e7f57f7c747a8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69992dc0903466da1e7f57f7c747a8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34167 } } }
I20260620 23:21:36.158254 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a69992dc0903466da1e7f57f7c747a8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69992dc0903466da1e7f57f7c747a8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34167 } } }
I20260620 23:21:36.158630 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:36.159228 2191 sys_catalog.cc:458] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:36.160113 2194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:21:36.165529 2194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:21:36.173810 2194 catalog_manager.cc:1347] Generated new cluster ID: c62004109c5c48b3bed97beadb83cfbd
I20260620 23:21:36.173906 2194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:21:36.194913 2194 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:21:36.196772 2194 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:21:36.213943 2194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f: Generated new TSK 0
I20260620 23:21:36.214994 2194 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:21:36.260533 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34167
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:36.586499 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:36.586895 2210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:36.587096 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:36.594476 2210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:36.594591 2210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:36.594835 2210 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:21:36.594931 2210 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:21:36.595057 2210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:21:36.604844 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:36.606156 2210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:36.607564 2210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:36.620790 2218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:36.620818 2216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:36.621165 2215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:36.621374 2210 server_base.cc:1034] running on GCE node
I20260620 23:21:36.622082 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:36.623067 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:36.624303 2210 hybrid_clock.cc:648] HybridClock initialized: now 1781997696624247 us; error 42 us; skew 500 ppm
I20260620 23:21:36.624626 2210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:36 dist-test-slave-5kzz krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:36.627686 2210 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:21:36.629140 2210 webserver.cc:460] Webserver started at http://127.0.97.1:34917/ using document root <none> and password file <none>
I20260620 23:21:36.629798 2210 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:36.629945 2210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:36.630259 2210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:36.633275 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "722fdf275bca4067b47b0fcd696c0c98"
format_stamp: "Formatted at 2026-06-20 23:21:36 on dist-test-slave-5kzz"
server_key: "c7db66cb1e160c4a7a540cd545dd7855"
server_key_iv: "b43d04245020310953c615b5c6acc858"
server_key_version: "encryptionkey@0"
I20260620 23:21:36.634070 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "722fdf275bca4067b47b0fcd696c0c98"
format_stamp: "Formatted at 2026-06-20 23:21:36 on dist-test-slave-5kzz"
server_key: "c7db66cb1e160c4a7a540cd545dd7855"
server_key_iv: "b43d04245020310953c615b5c6acc858"
server_key_version: "encryptionkey@0"
I20260620 23:21:36.639302 2210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 23:21:36.642601 2225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:36.643973 2210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:21:36.644096 2210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "722fdf275bca4067b47b0fcd696c0c98"
format_stamp: "Formatted at 2026-06-20 23:21:36 on dist-test-slave-5kzz"
server_key: "c7db66cb1e160c4a7a540cd545dd7855"
server_key_iv: "b43d04245020310953c615b5c6acc858"
server_key_version: "encryptionkey@0"
I20260620 23:21:36.644243 2210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:36.662757 2210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:36.666309 2210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:36.666527 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:36.668972 2210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:36.669106 2210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:36.669229 2210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:36.669317 2210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:21:36 dist-test-slave-5kzz krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:36.694144 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:36.670122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52501 (local address 127.0.97.62:34167)
0620 23:21:36.671085 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:36.671093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:36.671850 (+ 757us) server_negotiation.cc:407] Connection header received
0620 23:21:36.675198 (+ 3348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:36.675205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:36.675294 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:36.675449 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:36.677623 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:36.678605 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:36.681261 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:36.681645 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:36.681787 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:36.687381 (+ 5594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:36.687386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:36.687390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:36.687427 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:36.690812 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:36.691872 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:36.691880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:36.691886 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:36.692011 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:36.692457 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:36.692463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:36.692470 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:36.692731 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:36.692900 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:36.693581 (+ 681us) server_negotiation.cc:299] Negotiation successful
0620 23:21:36.693784 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":741,"threads_started":1}
I20260620 23:21:36.695528 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:36.670130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34167 (local address 127.0.97.1:52501)
0620 23:21:36.671666 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:36.671717 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:36.674824 (+ 3107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:36.675689 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:36.675696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:36.676482 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:36.677425 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:36.677458 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:36.678868 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:36.678875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:36.681078 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:36.681097 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:36.683082 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:36.683088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:36.683284 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:36.684150 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:36.684188 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:36.687137 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:36.691008 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:36.691019 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:36.691045 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:36.691690 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:36.692128 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:36.692133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:36.692137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:36.692331 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:36.692874 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:36.692883 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:36.693237 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 23:21:36.693496 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 23:21:36.694002 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1257,"thread_start_us":1015,"threads_started":1}
I20260620 23:21:36.714843 2210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35969
I20260620 23:21:36.714898 2345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35969 every 8 connection(s)
I20260620 23:21:36.716074 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:36.702555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37841 (local address 127.0.97.62:34167)
0620 23:21:36.702850 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:36.702856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:36.703443 (+ 587us) server_negotiation.cc:407] Connection header received
0620 23:21:36.703695 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:36.703702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:36.703796 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:36.703869 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:36.705397 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:36.706082 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:36.707450 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:36.707861 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:36.708007 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:36.710297 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:36.710303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:36.710308 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:36.710346 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:36.713257 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:36.713880 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:36.713888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:36.713894 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:36.714053 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:36.714599 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:36.714602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:36.714605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:36.714907 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:36.715061 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:36.715511 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 23:21:36.715713 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260620 23:21:36.716102 2210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:21:36.716172 2321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:36.702563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34167 (local address 127.0.97.1:37841)
0620 23:21:36.703325 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:36.703343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:36.703507 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:36.704062 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:36.704065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:36.704717 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:36.705248 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:36.705259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:36.706207 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:36.706215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:36.707314 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:36.707325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:36.708051 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:36.708057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:36.708231 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:36.708924 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:36.708941 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:36.710124 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:36.713387 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:36.713393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:36.713396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:36.713756 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:36.714255 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:36.714261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:36.714271 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:36.714455 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:36.715122 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:36.715126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:36.715269 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 23:21:36.715430 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 23:21:36.715721 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":483,"threads_started":1}
I20260620 23:21:36.718645 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 2210
I20260620 23:21:36.718854 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:21:36.719676 388 external_mini_cluster.cc:1428] Setting key edf14ce1343c2660507e26ff6ff7527f
I20260620 23:21:36.720355 2346 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34167
I20260620 23:21:36.720620 2346 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:36.721339 2346 heartbeater.cc:511] Master 127.0.97.62:34167 requested a full tablet report, sending...
I20260620 23:21:36.723579 2139 ts_manager.cc:194] Registered new tserver with Master: 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969)
I20260620 23:21:36.725616 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52501
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:21:36.757179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34167
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:37.070537 2350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:37.071130 2350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:37.071329 2350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:37.082223 2350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:37.082396 2350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:37.082567 2350 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:21:37.082646 2350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:21:37.082777 2350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:21:37.092674 2350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:37.093978 2350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:37.095393 2350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:37.108472 2356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:37.108688 2355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:37.109047 2358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:37.109681 2350 server_base.cc:1034] running on GCE node
I20260620 23:21:37.110291 2350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:37.111284 2350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:37.112498 2350 hybrid_clock.cc:648] HybridClock initialized: now 1781997697112440 us; error 58 us; skew 500 ppm
I20260620 23:21:37.112803 2350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:37 dist-test-slave-5kzz krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:37.116135 2350 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:21:37.123107 2350 webserver.cc:460] Webserver started at http://127.0.97.2:33217/ using document root <none> and password file <none>
I20260620 23:21:37.124064 2350 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:37.124192 2350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:37.124501 2350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:37.128273 2350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "0e540c1abb104d4a8c7ff3931c4f5938"
format_stamp: "Formatted at 2026-06-20 23:21:37 on dist-test-slave-5kzz"
server_key: "1d9741bae0b375c9f1104c6698bafd79"
server_key_iv: "050035899d23fe86d5c4e9b6151b9723"
server_key_version: "encryptionkey@0"
I20260620 23:21:37.129146 2350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0e540c1abb104d4a8c7ff3931c4f5938"
format_stamp: "Formatted at 2026-06-20 23:21:37 on dist-test-slave-5kzz"
server_key: "1d9741bae0b375c9f1104c6698bafd79"
server_key_iv: "050035899d23fe86d5c4e9b6151b9723"
server_key_version: "encryptionkey@0"
I20260620 23:21:37.136482 2350 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 23:21:37.140650 2365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:37.142158 2350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:21:37.142315 2350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "0e540c1abb104d4a8c7ff3931c4f5938"
format_stamp: "Formatted at 2026-06-20 23:21:37 on dist-test-slave-5kzz"
server_key: "1d9741bae0b375c9f1104c6698bafd79"
server_key_iv: "050035899d23fe86d5c4e9b6151b9723"
server_key_version: "encryptionkey@0"
I20260620 23:21:37.142544 2350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:37.156417 2350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:37.160614 2350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:37.160885 2350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:37.164098 2350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:37.164187 2350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:37.164270 2350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:37.164332 2350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:21:37 dist-test-slave-5kzz krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:37.196316 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.165344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34167 (local address 127.0.97.2:46655)
0620 23:21:37.166605 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:37.166663 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:37.169693 (+ 3030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:37.170938 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:37.170954 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:37.171946 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:37.173184 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.173221 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.175219 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.175226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.178176 (+ 2950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.178220 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.179701 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.179711 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.180119 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.181301 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:37.181335 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:37.185265 (+ 3930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:37.189884 (+ 4619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:37.189898 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:37.189924 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:37.190752 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:37.191271 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:37.191278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:37.191283 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:37.191565 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:37.192448 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:37.192462 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:37.193101 (+ 639us) client_negotiation.cc:769] Sending connection context
0620 23:21:37.193626 (+ 525us) client_negotiation.cc:240] Negotiation successful
0620 23:21:37.194374 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":446,"threads_started":1}
I20260620 23:21:37.197698 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.164724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46655 (local address 127.0.97.62:34167)
0620 23:21:37.164916 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:37.164924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:37.166824 (+ 1900us) server_negotiation.cc:407] Connection header received
0620 23:21:37.170090 (+ 3266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:37.170099 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:37.170207 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:37.170351 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:37.173448 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.174958 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.178469 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.179424 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.179600 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.185597 (+ 5997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:37.185606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:37.185615 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:37.185671 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:37.189698 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:37.190942 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:37.190948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:37.190951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:37.191125 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:37.191743 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:37.191750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:37.191756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:37.192170 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:37.192355 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:37.196936 (+ 4581us) server_negotiation.cc:299] Negotiation successful
0620 23:21:37.197270 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:21:37.226001 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.207546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38713 (local address 127.0.97.62:34167)
0620 23:21:37.207742 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:37.207751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:37.208417 (+ 666us) server_negotiation.cc:407] Connection header received
0620 23:21:37.208621 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:37.208630 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:37.208730 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:37.208865 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:37.210213 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.211654 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.214077 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.214966 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.215124 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.217790 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:37.217798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:37.217806 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:37.217858 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:37.222001 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:37.223116 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:37.223125 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:37.223153 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:37.223318 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:37.223932 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:37.223940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:37.223946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:37.224470 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:37.224640 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:37.225207 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 23:21:37.225505 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:21:37.226089 2440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.207153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34167 (local address 127.0.97.2:38713)
0620 23:21:37.208307 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:37.208329 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:37.208478 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:37.208954 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:37.208961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:37.209624 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:37.210039 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.210054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.211864 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.211872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.213859 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.213878 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.215196 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.215206 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.215423 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.216239 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:37.216265 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:37.217570 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:37.222174 (+ 4604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:37.222182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:37.222189 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:37.222918 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:37.223457 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:37.223464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:37.223470 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:37.223676 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:37.224639 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:37.224647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:37.224879 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 23:21:37.225057 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 23:21:37.225471 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"thread_start_us":832,"threads_started":1}
I20260620 23:21:37.239567 2350 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44889
I20260620 23:21:37.239607 2484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44889 every 8 connection(s)
I20260620 23:21:37.240763 2350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:21:37.243006 2485 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34167
I20260620 23:21:37.243235 2485 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:37.243872 2485 heartbeater.cc:511] Master 127.0.97.62:34167 requested a full tablet report, sending...
I20260620 23:21:37.244542 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 2350
I20260620 23:21:37.244719 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:21:37.245353 388 external_mini_cluster.cc:1428] Setting key 37bd6b90ca995fe3db3a664cb290d753
I20260620 23:21:37.246399 2139 ts_manager.cc:194] Registered new tserver with Master: 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889)
I20260620 23:21:37.247951 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46655
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:21:37.283742 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34167
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:37.602311 2489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:37.602626 2489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:37.602828 2489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:37.610129 2489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:37.610278 2489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:37.610441 2489 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:21:37.610529 2489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:21:37.610646 2489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:21:37.620393 2489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:37.621702 2489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:37.623081 2489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:37.635607 2497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:37.635725 2495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:37.635877 2494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:37.636618 2489 server_base.cc:1034] running on GCE node
I20260620 23:21:37.637207 2489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:37.638064 2489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:37.639293 2489 hybrid_clock.cc:648] HybridClock initialized: now 1781997697639252 us; error 49 us; skew 500 ppm
I20260620 23:21:37.639622 2489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:37 dist-test-slave-5kzz krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:37.646092 2489 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:21:37.647852 2489 webserver.cc:460] Webserver started at http://127.0.97.3:45483/ using document root <none> and password file <none>
I20260620 23:21:37.648675 2489 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:37.648816 2489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:37.649118 2489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:37.652256 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "733dd09c7b284d9fa3cb994a1c9b6e59"
format_stamp: "Formatted at 2026-06-20 23:21:37 on dist-test-slave-5kzz"
server_key: "715f29b942bd6c0f2ce5b7067b01ca58"
server_key_iv: "509cf5446b1b0913bdd6abcc9f452ae5"
server_key_version: "encryptionkey@0"
I20260620 23:21:37.653055 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "733dd09c7b284d9fa3cb994a1c9b6e59"
format_stamp: "Formatted at 2026-06-20 23:21:37 on dist-test-slave-5kzz"
server_key: "715f29b942bd6c0f2ce5b7067b01ca58"
server_key_iv: "509cf5446b1b0913bdd6abcc9f452ae5"
server_key_version: "encryptionkey@0"
I20260620 23:21:37.658293 2489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:21:37.661835 2504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:37.663210 2489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:21:37.663370 2489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "733dd09c7b284d9fa3cb994a1c9b6e59"
format_stamp: "Formatted at 2026-06-20 23:21:37 on dist-test-slave-5kzz"
server_key: "715f29b942bd6c0f2ce5b7067b01ca58"
server_key_iv: "509cf5446b1b0913bdd6abcc9f452ae5"
server_key_version: "encryptionkey@0"
I20260620 23:21:37.663570 2489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:37.684123 2489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:37.688398 2489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:37.688691 2489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:37.691134 2489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:37.691288 2489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:37.691413 2489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:37.691515 2489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:21:37 dist-test-slave-5kzz krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:37.714886 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.692443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34167 (local address 127.0.97.3:60475)
0620 23:21:37.693340 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:37.693392 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:37.695342 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:37.696106 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:37.696115 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:37.696980 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:37.697905 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.697947 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.699689 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.699697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.701839 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.701851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.703164 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.703170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.703390 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.704135 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:37.704161 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:37.706522 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:37.710003 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:37.710011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:37.710026 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:37.710462 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:37.710898 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:37.710905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:37.710909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:37.711191 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:37.711975 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:37.711981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:37.712485 (+ 504us) client_negotiation.cc:769] Sending connection context
0620 23:21:37.712853 (+ 368us) client_negotiation.cc:240] Negotiation successful
0620 23:21:37.713400 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":372,"threads_started":1}
I20260620 23:21:37.715775 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.691858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60475 (local address 127.0.97.62:34167)
0620 23:21:37.692065 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:37.692071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:37.693535 (+ 1464us) server_negotiation.cc:407] Connection header received
0620 23:21:37.695660 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:37.695666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:37.695751 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:37.695868 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:37.698141 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.699483 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.702006 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.702892 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.703067 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.706763 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:37.706768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:37.706771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:37.706801 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:37.709861 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:37.710587 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:37.710593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:37.710601 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:37.710783 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:37.711355 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:37.711360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:37.711363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:37.711765 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:37.711898 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:37.715233 (+ 3335us) server_negotiation.cc:299] Negotiation successful
0620 23:21:37.715454 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:21:37.728343 2346 heartbeater.cc:503] Master 127.0.97.62:34167 was elected leader, sending a full tablet report...
I20260620 23:21:37.740150 2570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.724115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34167 (local address 127.0.97.3:47213)
0620 23:21:37.724871 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:37.724902 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:37.725085 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:37.725429 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:37.725436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:37.726201 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:37.726658 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.726698 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.727967 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.727972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.729213 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.729228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.730493 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.730497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.730624 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.731493 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:37.731516 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:37.732357 (+ 841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:37.736953 (+ 4596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:37.736960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:37.736966 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:37.737491 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:37.737934 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:37.737940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:37.737946 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:37.738110 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:37.738785 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:37.738792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:37.738996 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 23:21:37.739209 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 23:21:37.739607 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":361,"threads_started":1}
I20260620 23:21:37.741562 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.724493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47213 (local address 127.0.97.62:34167)
0620 23:21:37.724810 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:37.724814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:37.724997 (+ 183us) server_negotiation.cc:407] Connection header received
0620 23:21:37.725213 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:37.725217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:37.725284 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:37.725364 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:37.726853 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.727813 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.729378 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.730307 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.730470 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.732559 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:37.732568 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:37.732576 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:37.732624 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:37.736709 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:37.737618 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:37.737625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:37.737631 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:37.737789 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:37.738224 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:37.738232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:37.738237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:37.738639 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:37.738821 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:37.740974 (+ 2153us) server_negotiation.cc:299] Negotiation successful
0620 23:21:37.741213 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260620 23:21:37.752357 2489 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44083
I20260620 23:21:37.752403 2623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44083 every 8 connection(s)
I20260620 23:21:37.753681 2489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:21:37.755623 2624 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34167
I20260620 23:21:37.755836 2624 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:37.756484 2624 heartbeater.cc:511] Master 127.0.97.62:34167 requested a full tablet report, sending...
I20260620 23:21:37.757714 2139 ts_manager.cc:194] Registered new tserver with Master: 733dd09c7b284d9fa3cb994a1c9b6e59 (127.0.97.3:44083)
I20260620 23:21:37.758529 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60475
I20260620 23:21:37.763288 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 2489
I20260620 23:21:37.763453 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:21:37.763856 388 external_mini_cluster.cc:1428] Setting key 5b7503936897462506cf9d2c512be072
I20260620 23:21:37.766407 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:21:37 dist-test-slave-5kzz krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:21:37 dist-test-slave-5kzz krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:37.810822 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.797314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43178 (local address 127.0.97.62:34167)
0620 23:21:37.797528 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:37.797535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:37.797731 (+ 196us) server_negotiation.cc:407] Connection header received
0620 23:21:37.797950 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:37.797959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:37.798043 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:37.798165 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:37.799508 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.800367 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.801645 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.802063 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.802207 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.805819 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:37.805823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:37.805826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:37.805857 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:37.808276 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:37.808866 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:37.808873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:37.808876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:37.809008 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:37.809370 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:37.809377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:37.809380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:37.809643 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:37.809874 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:37.810179 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 23:21:37.810398 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:21:37.814935 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:21:37.821843 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:21:37.848857 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.833927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44083 (local address 127.0.0.1:57830)
0620 23:21:37.835104 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:37.835123 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:37.835235 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:37.839530 (+ 4295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:37.839542 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:37.839583 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:37.840066 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.840082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.843898 (+ 3816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.843902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.846058 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.846075 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.847853 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.847860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.848069 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.848128 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 23:21:37.848274 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 23:21:37.848454 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"thread_start_us":8313,"threads_started":1}
I20260620 23:21:37.848891 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.834611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57830 (local address 127.0.97.3:44083)
0620 23:21:37.838653 (+ 4042us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:37.838666 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:37.838748 (+ 82us) server_negotiation.cc:407] Connection header received
0620 23:21:37.838872 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:37.838883 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:37.839218 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:37.840328 (+ 1110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:37.842282 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.843734 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.846239 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.847576 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.847797 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.848070 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:37.848316 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 23:21:37.848477 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3820,"thread_start_us":3604,"threads_started":1}
I20260620 23:21:37.851092 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.833466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35969 (local address 127.0.0.1:44802)
0620 23:21:37.834167 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:37.834212 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:37.834368 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:37.843440 (+ 9072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:37.843445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:37.843470 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:37.843905 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.843916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.845767 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.845772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.848173 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.848186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.850278 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.850283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.850462 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.850491 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:21:37.850554 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:21:37.850668 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":23,"thread_start_us":301,"threads_started":1}
I20260620 23:21:37.851495 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.833666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.0.97.1:35969)
0620 23:21:37.842896 (+ 9230us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:37.842907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:37.842933 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:21:37.843046 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:37.843052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:37.843229 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:37.843434 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:37.844073 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.845626 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.848352 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.850062 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.850785 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.850975 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:37.851040 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 23:21:37.851194 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9062,"thread_start_us":8935,"threads_started":1}
I20260620 23:21:37.851643 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.835044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.0.97.2:44889)
0620 23:21:37.842067 (+ 7023us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:37.842075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:37.842628 (+ 553us) server_negotiation.cc:407] Connection header received
0620 23:21:37.842991 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:37.842999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:37.843230 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:37.843438 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:37.844352 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.845819 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.848363 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.849750 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.849876 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.850064 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:37.851189 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0620 23:21:37.851379 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6889,"thread_start_us":6796,"threads_started":1}
I20260620 23:21:37.852074 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:37.834517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44889 (local address 127.0.0.1:45168)
0620 23:21:37.842513 (+ 7996us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:37.842533 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:37.842758 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:37.843640 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:37.843645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:37.843676 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:37.844161 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.844176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.845952 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.845962 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.848184 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:37.848193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:37.850224 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:37.850231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:37.850816 (+ 585us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:37.850852 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:21:37.850937 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:21:37.851062 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7848,"mutex_wait_us":383,"thread_start_us":650,"threads_started":1}
I20260620 23:21:37.853183 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 3df05d8c08694d7b818ced1ebf023ac2 (DEFAULT_TABLE table=test-table [id=c85c8d56c0ae4655a2d1fa370770384c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:37.854794 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df05d8c08694d7b818ced1ebf023ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:37.854843 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 3df05d8c08694d7b818ced1ebf023ac2 (DEFAULT_TABLE table=test-table [id=c85c8d56c0ae4655a2d1fa370770384c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:37.855382 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 3df05d8c08694d7b818ced1ebf023ac2 (DEFAULT_TABLE table=test-table [id=c85c8d56c0ae4655a2d1fa370770384c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:37.856276 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df05d8c08694d7b818ced1ebf023ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:37.856599 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df05d8c08694d7b818ced1ebf023ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:37.868184 2643 tablet_bootstrap.cc:492] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98: Bootstrap starting.
I20260620 23:21:37.870795 2645 tablet_bootstrap.cc:492] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938: Bootstrap starting.
I20260620 23:21:37.871760 2643 tablet_bootstrap.cc:654] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:37.872424 2644 tablet_bootstrap.cc:492] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59: Bootstrap starting.
I20260620 23:21:37.873114 2643 log.cc:824] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:37.874322 2645 tablet_bootstrap.cc:654] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:37.875536 2645 log.cc:824] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:37.875936 2644 tablet_bootstrap.cc:654] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:37.876816 2644 log.cc:824] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:37.877389 2643 tablet_bootstrap.cc:492] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98: No bootstrap required, opened a new log
I20260620 23:21:37.877745 2643 ts_tablet_manager.cc:1397] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 23:21:37.879274 2644 tablet_bootstrap.cc:492] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59: No bootstrap required, opened a new log
I20260620 23:21:37.879560 2644 ts_tablet_manager.cc:1397] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 23:21:37.884461 2645 tablet_bootstrap.cc:492] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938: No bootstrap required, opened a new log
I20260620 23:21:37.884719 2645 ts_tablet_manager.cc:1397] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260620 23:21:37.887339 2644 raft_consensus.cc:348] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:37.887686 2644 raft_consensus.cc:374] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:37.887782 2644 raft_consensus.cc:729] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733dd09c7b284d9fa3cb994a1c9b6e59, State: Initialized, Role: FOLLOWER
I20260620 23:21:37.888372 2644 consensus_queue.cc:260] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:37.889367 2624 heartbeater.cc:503] Master 127.0.97.62:34167 was elected leader, sending a full tablet report...
I20260620 23:21:37.889338 2643 raft_consensus.cc:348] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:37.889739 2643 raft_consensus.cc:374] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:37.889833 2643 raft_consensus.cc:729] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722fdf275bca4067b47b0fcd696c0c98, State: Initialized, Role: FOLLOWER
I20260620 23:21:37.890383 2643 consensus_queue.cc:260] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:37.891332 2644 ts_tablet_manager.cc:1428] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 23:21:37.896739 2645 raft_consensus.cc:348] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:37.897320 2645 raft_consensus.cc:374] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:37.897452 2645 raft_consensus.cc:729] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e540c1abb104d4a8c7ff3931c4f5938, State: Initialized, Role: FOLLOWER
I20260620 23:21:37.898178 2645 consensus_queue.cc:260] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:37.899273 2485 heartbeater.cc:503] Master 127.0.97.62:34167 was elected leader, sending a full tablet report...
I20260620 23:21:37.900002 2643 ts_tablet_manager.cc:1428] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
I20260620 23:21:37.901090 2645 ts_tablet_manager.cc:1428] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260620 23:21:37.968748 2347 tablet.cc:2369] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:21:37.993136 2486 tablet.cc:2369] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:21:38.005326 2625 tablet.cc:2369] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:21:38.233425 2649 raft_consensus.cc:482] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:21:38.233713 2649 raft_consensus.cc:504] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.235378 2649 leader_election.cc:288] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969), 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889)
I20260620 23:21:38.242544 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:38.236619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40795 (local address 127.0.97.2:44889)
0620 23:21:38.236801 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:38.236806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:38.237015 (+ 209us) server_negotiation.cc:407] Connection header received
0620 23:21:38.237124 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:38.237131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:38.237204 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:38.237305 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:38.237832 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.238539 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.240323 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.241617 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.241756 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:38.241869 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:38.242150 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:21:38.242276 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:21:38.242820 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:38.235834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49771 (local address 127.0.97.1:35969)
0620 23:21:38.236081 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:38.236089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:38.236412 (+ 323us) server_negotiation.cc:407] Connection header received
0620 23:21:38.236654 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:38.236660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:38.236735 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:38.236798 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:38.237596 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.238309 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.240720 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.241934 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.242072 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:38.242176 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:38.242446 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:21:38.242567 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:21:38.242946 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:38.235684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35969 (local address 127.0.97.3:49771)
0620 23:21:38.236249 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:38.236265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:38.236463 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:38.237015 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:38.237019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:38.237049 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:38.237394 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:38.237408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.238454 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.238459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:38.240549 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:38.240565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.242131 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.242137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:38.242297 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:38.242318 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:21:38.242421 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 23:21:38.242546 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":2,"thread_start_us":321,"threads_started":1}
I20260620 23:21:38.243347 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:38.236401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44889 (local address 127.0.97.3:40795)
0620 23:21:38.236913 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:38.236927 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:38.237029 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:38.237321 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:38.237324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:38.237343 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:38.237680 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:38.237686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.238706 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.238712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:38.240173 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:38.240184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.241792 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.241798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:38.241963 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:38.241996 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:21:38.242772 (+ 776us) client_negotiation.cc:240] Negotiation successful
0620 23:21:38.242898 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":5,"thread_start_us":257,"threads_started":1}
I20260620 23:21:38.243916 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df05d8c08694d7b818ced1ebf023ac2" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722fdf275bca4067b47b0fcd696c0c98" is_pre_election: true
I20260620 23:21:38.244313 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df05d8c08694d7b818ced1ebf023ac2" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" is_pre_election: true
I20260620 23:21:38.244488 2296 raft_consensus.cc:2454] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 0.
I20260620 23:21:38.244841 2439 raft_consensus.cc:2454] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 0.
I20260620 23:21:38.245139 2506 leader_election.cc:302] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722fdf275bca4067b47b0fcd696c0c98, 733dd09c7b284d9fa3cb994a1c9b6e59; no voters:
I20260620 23:21:38.245534 2649 raft_consensus.cc:2781] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:21:38.245684 2649 raft_consensus.cc:482] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:21:38.245795 2649 raft_consensus.cc:3037] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:38.247987 2649 raft_consensus.cc:504] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.248607 2649 leader_election.cc:288] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 election: Requested vote from peers 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969), 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889)
I20260620 23:21:38.248983 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df05d8c08694d7b818ced1ebf023ac2" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722fdf275bca4067b47b0fcd696c0c98"
I20260620 23:21:38.249109 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df05d8c08694d7b818ced1ebf023ac2" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e540c1abb104d4a8c7ff3931c4f5938"
I20260620 23:21:38.249193 2296 raft_consensus.cc:3037] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:38.249320 2439 raft_consensus.cc:3037] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:38.251559 2296 raft_consensus.cc:2454] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 1.
I20260620 23:21:38.251925 2506 leader_election.cc:302] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722fdf275bca4067b47b0fcd696c0c98, 733dd09c7b284d9fa3cb994a1c9b6e59; no voters:
I20260620 23:21:38.252198 2649 raft_consensus.cc:2781] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:38.252527 2439 raft_consensus.cc:2454] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 1.
I20260620 23:21:38.252652 2649 raft_consensus.cc:686] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 LEADER]: Becoming Leader. State: Replica: 733dd09c7b284d9fa3cb994a1c9b6e59, State: Running, Role: LEADER
I20260620 23:21:38.253212 2649 consensus_queue.cc:237] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.257727 2138 catalog_manager.cc:5495] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 733dd09c7b284d9fa3cb994a1c9b6e59 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } health_report { overall_health: HEALTHY } } }
Jun 20 23:21:38 dist-test-slave-5kzz krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:38.321468 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:38.309101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34167 (local address 127.0.0.1:43192)
0620 23:21:38.309697 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:38.309713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:38.309872 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:38.310156 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:38.310159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:38.310910 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:38.311297 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:38.311312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.312247 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.312251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:38.313201 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:38.313211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.313863 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.313867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:38.313992 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:38.314587 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:38.314603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:38.316705 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:38.319230 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:38.319239 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:38.319245 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:38.319657 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:38.319994 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:38.319997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:38.319999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:38.320121 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:38.320537 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:38.320543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:38.320690 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 23:21:38.320852 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 23:21:38.321124 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":343,"threads_started":1}
I20260620 23:21:38.321512 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:38.308994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.0.97.62:34167)
0620 23:21:38.309190 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:38.309196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:38.309832 (+ 636us) server_negotiation.cc:407] Connection header received
0620 23:21:38.309968 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:38.309972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:38.310029 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:38.310112 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:38.311463 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.312137 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.313312 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.313716 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.313803 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:38.316855 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:38.316859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:38.316862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:38.316889 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:38.319121 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:38.319751 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:38.319754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:38.319756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:38.319878 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:38.320208 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:38.320211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:38.320213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:38.320440 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:38.320539 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:38.320952 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 23:21:38.321241 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":69}
I20260620 23:21:38.326316 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43192:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:21:38.327302 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:21:38.333526 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 88aee5d7d7c24c49b3e3c63f79cb39a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=537563293aa549d8803f5ce165b69ea4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:21:38.333526 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 88aee5d7d7c24c49b3e3c63f79cb39a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=537563293aa549d8803f5ce165b69ea4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:21:38.333739 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 88aee5d7d7c24c49b3e3c63f79cb39a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=537563293aa549d8803f5ce165b69ea4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:21:38.334501 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88aee5d7d7c24c49b3e3c63f79cb39a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:38.334496 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88aee5d7d7c24c49b3e3c63f79cb39a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:38.334496 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88aee5d7d7c24c49b3e3c63f79cb39a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:38.340070 2644 tablet_bootstrap.cc:492] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: Bootstrap starting.
I20260620 23:21:38.341419 2644 tablet_bootstrap.cc:654] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:38.341495 2643 tablet_bootstrap.cc:492] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98: Bootstrap starting.
I20260620 23:21:38.341768 2645 tablet_bootstrap.cc:492] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938: Bootstrap starting.
I20260620 23:21:38.343147 2643 tablet_bootstrap.cc:654] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:38.343297 2645 tablet_bootstrap.cc:654] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:38.344074 2644 tablet_bootstrap.cc:492] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: No bootstrap required, opened a new log
I20260620 23:21:38.344249 2644 ts_tablet_manager.cc:1397] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:21:38.344921 2644 raft_consensus.cc:348] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.345175 2644 raft_consensus.cc:374] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:38.345271 2644 raft_consensus.cc:729] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733dd09c7b284d9fa3cb994a1c9b6e59, State: Initialized, Role: FOLLOWER
I20260620 23:21:38.345521 2645 tablet_bootstrap.cc:492] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938: No bootstrap required, opened a new log
I20260620 23:21:38.345623 2643 tablet_bootstrap.cc:492] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98: No bootstrap required, opened a new log
I20260620 23:21:38.345669 2645 ts_tablet_manager.cc:1397] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:21:38.345541 2644 consensus_queue.cc:260] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.345765 2643 ts_tablet_manager.cc:1397] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:21:38.345960 2649 tablet_replica.cc:439] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } } }
I20260620 23:21:38.346239 2649 tablet_replica.cc:442] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:38.346480 2645 raft_consensus.cc:348] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.346529 2643 raft_consensus.cc:348] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.346755 2645 raft_consensus.cc:374] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:38.346760 2643 raft_consensus.cc:374] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:38.346853 2643 raft_consensus.cc:729] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722fdf275bca4067b47b0fcd696c0c98, State: Initialized, Role: FOLLOWER
I20260620 23:21:38.346856 2645 raft_consensus.cc:729] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e540c1abb104d4a8c7ff3931c4f5938, State: Initialized, Role: FOLLOWER
I20260620 23:21:38.347081 2643 consensus_queue.cc:260] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.347074 2645 consensus_queue.cc:260] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.347671 2645 ts_tablet_manager.cc:1428] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 23:21:38.347745 2643 ts_tablet_manager.cc:1428] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:21:38.347600 2650 tablet_replica.cc:439] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } } }
I20260620 23:21:38.347965 2650 tablet_replica.cc:442] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:38.348050 2651 tablet_replica.cc:439] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } } }
I20260620 23:21:38.348400 2651 tablet_replica.cc:442] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:38.349793 2644 ts_tablet_manager.cc:1428] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 23:21:38.579531 2649 raft_consensus.cc:482] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:21:38.579730 2649 raft_consensus.cc:504] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.580281 2649 leader_election.cc:288] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969), 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889)
I20260620 23:21:38.580875 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88aee5d7d7c24c49b3e3c63f79cb39a7" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722fdf275bca4067b47b0fcd696c0c98" is_pre_election: true
I20260620 23:21:38.580972 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88aee5d7d7c24c49b3e3c63f79cb39a7" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" is_pre_election: true
I20260620 23:21:38.581172 2296 raft_consensus.cc:2454] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 0.
I20260620 23:21:38.581238 2439 raft_consensus.cc:2454] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 0.
I20260620 23:21:38.581589 2506 leader_election.cc:302] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722fdf275bca4067b47b0fcd696c0c98, 733dd09c7b284d9fa3cb994a1c9b6e59; no voters:
I20260620 23:21:38.581887 2649 raft_consensus.cc:2781] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:21:38.581974 2649 raft_consensus.cc:482] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:21:38.582094 2649 raft_consensus.cc:3037] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:38.584075 2649 raft_consensus.cc:504] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.584640 2649 leader_election.cc:288] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 election: Requested vote from peers 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969), 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889)
I20260620 23:21:38.585003 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88aee5d7d7c24c49b3e3c63f79cb39a7" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722fdf275bca4067b47b0fcd696c0c98"
I20260620 23:21:38.585105 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88aee5d7d7c24c49b3e3c63f79cb39a7" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e540c1abb104d4a8c7ff3931c4f5938"
I20260620 23:21:38.585224 2296 raft_consensus.cc:3037] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:38.585314 2439 raft_consensus.cc:3037] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:38.588092 2439 raft_consensus.cc:2454] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 1.
I20260620 23:21:38.588092 2296 raft_consensus.cc:2454] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 1.
I20260620 23:21:38.588487 2506 leader_election.cc:302] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722fdf275bca4067b47b0fcd696c0c98, 733dd09c7b284d9fa3cb994a1c9b6e59; no voters:
I20260620 23:21:38.588770 2649 raft_consensus.cc:2781] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:38.588941 2649 raft_consensus.cc:686] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 LEADER]: Becoming Leader. State: Replica: 733dd09c7b284d9fa3cb994a1c9b6e59, State: Running, Role: LEADER
I20260620 23:21:38.589180 2649 consensus_queue.cc:237] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.590695 2654 tablet_replica.cc:439] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: TxnStatusTablet state changed. Reason: New leader 733dd09c7b284d9fa3cb994a1c9b6e59. Latest consensus state: current_term: 1 leader_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } } }
I20260620 23:21:38.590977 2654 tablet_replica.cc:442] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:21:38.591540 2666 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:21:38.591809 2666 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:21:38.592072 2137 catalog_manager.cc:5495] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 733dd09c7b284d9fa3cb994a1c9b6e59 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } health_report { overall_health: HEALTHY } } }
I20260620 23:21:38.641247 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:38.634954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44083 (local address 127.0.0.1:57834)
0620 23:21:38.635233 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:38.635260 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:38.635448 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:38.636442 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:38.636447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:38.636458 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:21:38.636664 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:38.636675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.637742 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.637748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:38.638801 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:38.638812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.639586 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.639591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:38.639771 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:38.639822 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:21:38.640722 (+ 900us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:21:38.640743 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:21:38.640820 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 23:21:38.640964 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 23:21:38.641311 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:38.634985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.0.97.3:44083)
0620 23:21:38.636078 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:38.636082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:38.636100 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:21:38.636170 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:38.636176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:38.636246 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:38.636425 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:21:38.636858 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.637615 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.638940 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.639390 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.639556 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:38.640045 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:21:38.640601 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:21:38.640752 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:38.640854 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 23:21:38.640925 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":813,"threads_started":1}
I20260620 23:21:38.648862 2439 raft_consensus.cc:1261] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Refusing update from remote peer 733dd09c7b284d9fa3cb994a1c9b6e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:38.648900 2296 raft_consensus.cc:1261] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Refusing update from remote peer 733dd09c7b284d9fa3cb994a1c9b6e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:38.649875 2654 consensus_queue.cc:1035] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:38.650240 2649 consensus_queue.cc:1035] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:38.660750 2651 tablet_replica.cc:439] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938: TxnStatusTablet state changed. Reason: New leader 733dd09c7b284d9fa3cb994a1c9b6e59. Latest consensus state: current_term: 1 leader_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } } }
I20260620 23:21:38.661054 2651 tablet_replica.cc:442] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:38.663811 2670 mvcc.cc:205] Tried to move back new op lower bound from 7299062573655764992 to 7299062573423349760. Current Snapshot: MvccSnapshot[applied={T|T < 7299062573655764992}]
I20260620 23:21:38.665307 2654 tablet_replica.cc:439] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } } }
I20260620 23:21:38.665606 2654 tablet_replica.cc:442] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:21:38.669165 2651 tablet_replica.cc:439] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } } }
I20260620 23:21:38.669396 2651 tablet_replica.cc:442] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 0e540c1abb104d4a8c7ff3931c4f5938: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:38.675863 2649 tablet_replica.cc:439] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } } }
I20260620 23:21:38.676282 2649 tablet_replica.cc:442] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 733dd09c7b284d9fa3cb994a1c9b6e59: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:21:38.677008 2650 tablet_replica.cc:439] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98: TxnStatusTablet state changed. Reason: New leader 733dd09c7b284d9fa3cb994a1c9b6e59. Latest consensus state: current_term: 1 leader_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } } }
I20260620 23:21:38.677343 2650 tablet_replica.cc:442] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:38.682823 2672 mvcc.cc:205] Tried to move back new op lower bound from 7299062573655764992 to 7299062573423349760. Current Snapshot: MvccSnapshot[applied={T|T < 7299062573655764992}]
I20260620 23:21:38.683662 2650 tablet_replica.cc:439] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } } }
I20260620 23:21:38.683868 2650 tablet_replica.cc:442] T 88aee5d7d7c24c49b3e3c63f79cb39a7 P 722fdf275bca4067b47b0fcd696c0c98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:21:38.692178 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:38.686746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.0.97.3:44083)
0620 23:21:38.686894 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:38.686898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:38.687415 (+ 517us) server_negotiation.cc:407] Connection header received
0620 23:21:38.687618 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:38.687623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:38.687696 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:38.687786 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:21:38.688235 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.688990 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.690361 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.690879 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.691015 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:38.691401 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:21:38.691502 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:21:38.691610 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:38.691749 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 23:21:38.691822 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:21:38.709764 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:38.703413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52347 (local address 127.0.97.3:44083)
0620 23:21:38.703600 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:38.703608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:38.704242 (+ 634us) server_negotiation.cc:407] Connection header received
0620 23:21:38.704430 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:38.704433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:38.704498 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:38.704573 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:21:38.705145 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.705901 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.708058 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.708473 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.708569 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:38.709113 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:21:38.709204 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:21:38.709264 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:38.709424 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 23:21:38.709480 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:21:38.710186 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:38.703419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44083 (local address 127.0.97.3:52347)
0620 23:21:38.704139 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:38.704158 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:38.704303 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:38.704610 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:38.704615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:38.704630 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:21:38.705007 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:38.705020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.706035 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.706041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:38.707927 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:38.707941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:38.708652 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:38.708659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:38.708850 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:38.708920 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:21:38.709301 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:21:38.709312 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:21:38.709410 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 23:21:38.709572 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":18,"thread_start_us":370,"threads_started":1}
I20260620 23:21:38.720199 2439 raft_consensus.cc:1261] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Refusing update from remote peer 733dd09c7b284d9fa3cb994a1c9b6e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:38.720209 2296 raft_consensus.cc:1261] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Refusing update from remote peer 733dd09c7b284d9fa3cb994a1c9b6e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:38.721278 2677 consensus_queue.cc:1035] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:38.721663 2649 consensus_queue.cc:1035] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:38.814940 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:38.815213 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:38.818854 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f: Sending DeleteTablet for 3 replicas of tablet 3df05d8c08694d7b818ced1ebf023ac2
I20260620 23:21:38.819972 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df05d8c08694d7b818ced1ebf023ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45168
I20260620 23:21:38.820020 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df05d8c08694d7b818ced1ebf023ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57830
I20260620 23:21:38.820930 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df05d8c08694d7b818ced1ebf023ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44802
I20260620 23:21:38.821295 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:21:38.822152 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:21:38.828866 2689 tablet_replica.cc:330] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938: stopping tablet replica
I20260620 23:21:38.829958 2689 raft_consensus.cc:2229] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:38.831017 2689 raft_consensus.cc:2258] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:38.831383 2553 tablet_service.cc:1467] Processing CreateTablet for tablet cf95e386f7f5418ca012d323830cf37e (DEFAULT_TABLE table=test-table [id=c82bb018e7b046cca2d29c8184ecf3a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:38.832101 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf95e386f7f5418ca012d323830cf37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:38.835650 2691 tablet_replica.cc:330] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98: stopping tablet replica
I20260620 23:21:38.836609 2415 tablet_service.cc:1467] Processing CreateTablet for tablet cf95e386f7f5418ca012d323830cf37e (DEFAULT_TABLE table=test-table [id=c82bb018e7b046cca2d29c8184ecf3a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:38.837023 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf95e386f7f5418ca012d323830cf37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:38.837863 2689 ts_tablet_manager.cc:1905] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:38.841883 2644 tablet_bootstrap.cc:492] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59: Bootstrap starting.
I20260620 23:21:38.842552 2276 tablet_service.cc:1467] Processing CreateTablet for tablet cf95e386f7f5418ca012d323830cf37e (DEFAULT_TABLE table=test-table [id=c82bb018e7b046cca2d29c8184ecf3a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:38.843083 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf95e386f7f5418ca012d323830cf37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:38.843660 2644 tablet_bootstrap.cc:654] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:38.844027 2645 tablet_bootstrap.cc:492] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938: Bootstrap starting.
I20260620 23:21:38.845597 2645 tablet_bootstrap.cc:654] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:38.850806 2691 raft_consensus.cc:2229] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:38.851369 2691 raft_consensus.cc:2258] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:38.852931 2690 tablet_replica.cc:330] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59: stopping tablet replica
I20260620 23:21:38.853447 2691 ts_tablet_manager.cc:1905] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:38.853637 2690 raft_consensus.cc:2229] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:21:38.854491 2690 raft_consensus.cc:2258] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:38.858285 2694 tablet_bootstrap.cc:492] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98: Bootstrap starting.
I20260620 23:21:38.859373 2645 tablet_bootstrap.cc:492] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938: No bootstrap required, opened a new log
I20260620 23:21:38.859558 2645 ts_tablet_manager.cc:1397] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.006s
I20260620 23:21:38.860288 2645 raft_consensus.cc:348] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.860523 2645 raft_consensus.cc:374] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:38.860584 2645 raft_consensus.cc:729] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e540c1abb104d4a8c7ff3931c4f5938, State: Initialized, Role: FOLLOWER
I20260620 23:21:38.860711 2694 tablet_bootstrap.cc:654] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:38.861024 2645 consensus_queue.cc:260] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.863830 2645 ts_tablet_manager.cc:1428] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 23:21:38.862771 2690 ts_tablet_manager.cc:1905] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:38.866967 2691 ts_tablet_manager.cc:1918] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:21:38.868201 2644 tablet_bootstrap.cc:492] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59: No bootstrap required, opened a new log
I20260620 23:21:38.868433 2644 ts_tablet_manager.cc:1397] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.009s
I20260620 23:21:38.869342 2644 raft_consensus.cc:348] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.869649 2644 raft_consensus.cc:374] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:38.869760 2644 raft_consensus.cc:729] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733dd09c7b284d9fa3cb994a1c9b6e59, State: Initialized, Role: FOLLOWER
I20260620 23:21:38.870074 2644 consensus_queue.cc:260] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.873158 2644 ts_tablet_manager.cc:1428] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260620 23:21:38.875401 2691 log.cc:1196] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/3df05d8c08694d7b818ced1ebf023ac2
I20260620 23:21:38.875994 2691 ts_tablet_manager.cc:1939] T 3df05d8c08694d7b818ced1ebf023ac2 P 722fdf275bca4067b47b0fcd696c0c98: Deleting consensus metadata
I20260620 23:21:38.876715 2689 ts_tablet_manager.cc:1918] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:21:38.877061 2689 log.cc:1196] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/3df05d8c08694d7b818ced1ebf023ac2
I20260620 23:21:38.877753 2689 ts_tablet_manager.cc:1939] T 3df05d8c08694d7b818ced1ebf023ac2 P 0e540c1abb104d4a8c7ff3931c4f5938: Deleting consensus metadata
I20260620 23:21:38.877851 2124 catalog_manager.cc:4841] TS 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969): tablet 3df05d8c08694d7b818ced1ebf023ac2 (table test-table [id=c85c8d56c0ae4655a2d1fa370770384c]) successfully deleted
I20260620 23:21:38.879505 2124 catalog_manager.cc:4841] TS 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889): tablet 3df05d8c08694d7b818ced1ebf023ac2 (table test-table [id=c85c8d56c0ae4655a2d1fa370770384c]) successfully deleted
I20260620 23:21:38.879626 2694 tablet_bootstrap.cc:492] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98: No bootstrap required, opened a new log
I20260620 23:21:38.879830 2694 ts_tablet_manager.cc:1397] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.012s
I20260620 23:21:38.880491 2694 raft_consensus.cc:348] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.880708 2694 raft_consensus.cc:374] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:38.880864 2694 raft_consensus.cc:729] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722fdf275bca4067b47b0fcd696c0c98, State: Initialized, Role: FOLLOWER
I20260620 23:21:38.881165 2694 consensus_queue.cc:260] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:38.881687 2694 ts_tablet_manager.cc:1428] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 23:21:38.882206 2690 ts_tablet_manager.cc:1918] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:21:38.882390 2690 log.cc:1196] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/3df05d8c08694d7b818ced1ebf023ac2
I20260620 23:21:38.882963 2690 ts_tablet_manager.cc:1939] T 3df05d8c08694d7b818ced1ebf023ac2 P 733dd09c7b284d9fa3cb994a1c9b6e59: Deleting consensus metadata
I20260620 23:21:38.884591 2125 catalog_manager.cc:4841] TS 733dd09c7b284d9fa3cb994a1c9b6e59 (127.0.97.3:44083): tablet 3df05d8c08694d7b818ced1ebf023ac2 (table test-table [id=c85c8d56c0ae4655a2d1fa370770384c]) successfully deleted
I20260620 23:21:39.035569 2651 raft_consensus.cc:482] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:21:39.035836 2651 raft_consensus.cc:504] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:39.037253 2651 leader_election.cc:288] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969), 733dd09c7b284d9fa3cb994a1c9b6e59 (127.0.97.3:44083)
I20260620 23:21:39.046716 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:39.037942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47125 (local address 127.0.97.1:35969)
0620 23:21:39.039289 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:39.039294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:39.039318 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:21:39.039401 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:39.039405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:39.039466 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:39.039575 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:39.040250 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.041395 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.044273 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.045577 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.045706 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:39.045818 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:39.046206 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 23:21:39.046360 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1173,"thread_start_us":1004,"threads_started":1}
I20260620 23:21:39.046743 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:39.037618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55467 (local address 127.0.97.3:44083)
0620 23:21:39.037937 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:39.037944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:39.039575 (+ 1631us) server_negotiation.cc:407] Connection header received
0620 23:21:39.039874 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:39.039881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:39.039963 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:39.040064 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:39.040673 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.041688 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.044106 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.045599 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.045706 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:39.045819 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:39.046351 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 23:21:39.046494 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260620 23:21:39.046802 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:39.037538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35969 (local address 127.0.97.2:47125)
0620 23:21:39.038467 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:39.038492 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:39.038700 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:39.039607 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:39.039615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:39.039643 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:39.040015 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:39.040029 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.041560 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.041566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:39.044121 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:39.044135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.045729 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.045736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:39.046021 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:39.046065 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 23:21:39.046170 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:21:39.046306 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":62,"thread_start_us":576,"threads_started":1}
I20260620 23:21:39.047389 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf95e386f7f5418ca012d323830cf37e" candidate_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722fdf275bca4067b47b0fcd696c0c98" is_pre_election: true
I20260620 23:21:39.047680 2296 raft_consensus.cc:2454] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e540c1abb104d4a8c7ff3931c4f5938 in term 0.
I20260620 23:21:39.047721 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:39.037601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44083 (local address 127.0.97.2:55467)
0620 23:21:39.039479 (+ 1878us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:39.039497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:39.039680 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:39.040124 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:39.040132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:39.040166 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:39.040493 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:39.040506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.041811 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.041818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:39.043963 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:39.043975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.046024 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.046036 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:39.046184 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:39.046215 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:21:39.047204 (+ 989us) client_negotiation.cc:240] Negotiation successful
0620 23:21:39.047355 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1767,"mutex_wait_us":76,"spinlock_wait_cycles":3840,"thread_start_us":1582,"threads_started":1}
I20260620 23:21:39.048203 2367 leader_election.cc:302] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e540c1abb104d4a8c7ff3931c4f5938, 722fdf275bca4067b47b0fcd696c0c98; no voters:
I20260620 23:21:39.048703 2651 raft_consensus.cc:2781] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:21:39.048588 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf95e386f7f5418ca012d323830cf37e" candidate_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" is_pre_election: true
I20260620 23:21:39.048818 2651 raft_consensus.cc:482] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:21:39.048904 2651 raft_consensus.cc:3037] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:39.049063 2579 raft_consensus.cc:2454] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e540c1abb104d4a8c7ff3931c4f5938 in term 0.
I20260620 23:21:39.051255 2651 raft_consensus.cc:504] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:39.051854 2651 leader_election.cc:288] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [CANDIDATE]: Term 1 election: Requested vote from peers 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969), 733dd09c7b284d9fa3cb994a1c9b6e59 (127.0.97.3:44083)
I20260620 23:21:39.052266 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf95e386f7f5418ca012d323830cf37e" candidate_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722fdf275bca4067b47b0fcd696c0c98"
I20260620 23:21:39.052459 2296 raft_consensus.cc:3037] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:39.052390 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf95e386f7f5418ca012d323830cf37e" candidate_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59"
I20260620 23:21:39.052591 2579 raft_consensus.cc:3037] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:39.054814 2579 raft_consensus.cc:2454] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e540c1abb104d4a8c7ff3931c4f5938 in term 1.
I20260620 23:21:39.055249 2368 leader_election.cc:302] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e540c1abb104d4a8c7ff3931c4f5938, 733dd09c7b284d9fa3cb994a1c9b6e59; no voters:
I20260620 23:21:39.055436 2296 raft_consensus.cc:2454] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e540c1abb104d4a8c7ff3931c4f5938 in term 1.
I20260620 23:21:39.055516 2651 raft_consensus.cc:2781] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:39.056252 2651 raft_consensus.cc:686] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 LEADER]: Becoming Leader. State: Replica: 0e540c1abb104d4a8c7ff3931c4f5938, State: Running, Role: LEADER
I20260620 23:21:39.056627 2651 consensus_queue.cc:237] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:39.060269 2139 catalog_manager.cc:5495] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20260620 23:21:39.109041 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:39.102477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.0.97.2:44889)
0620 23:21:39.103250 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:39.103254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:39.103271 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:21:39.103342 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:39.103345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:39.103417 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:39.103557 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:21:39.103966 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.104711 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.106666 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.107185 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.107301 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:39.107783 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:21:39.108368 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:21:39.108473 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:39.108612 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 23:21:39.108681 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":547,"threads_started":1}
I20260620 23:21:39.124405 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:39.118482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46453 (local address 127.0.97.3:44083)
0620 23:21:39.118715 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:39.118724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:39.119378 (+ 654us) server_negotiation.cc:407] Connection header received
0620 23:21:39.119577 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:39.119582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:39.119675 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:39.119773 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:21:39.120321 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.121132 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.122516 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.123066 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.123226 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:39.123627 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:21:39.123724 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:21:39.123819 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:39.123995 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 23:21:39.124074 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:21:39.124464 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:39.118415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44083 (local address 127.0.97.2:46453)
0620 23:21:39.119265 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:39.119285 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:39.119453 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:39.119797 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:39.119802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:39.119812 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:21:39.120149 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:39.120162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.121329 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.121334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:39.122369 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:39.122380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.123246 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.123252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:39.123400 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:39.123455 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:21:39.123866 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:21:39.123882 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 23:21:39.123971 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:21:39.124104 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":518,"threads_started":1}
I20260620 23:21:39.134034 2296 raft_consensus.cc:1261] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Refusing update from remote peer 0e540c1abb104d4a8c7ff3931c4f5938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:39.134167 2579 raft_consensus.cc:1261] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Refusing update from remote peer 0e540c1abb104d4a8c7ff3931c4f5938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:39.134716 2699 consensus_queue.cc:1035] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:39.135406 2699 consensus_queue.cc:1035] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:39.147173 2671 mvcc.cc:205] Tried to move back new op lower bound from 7299062575647043584 to 7299062575339778048. Current Snapshot: MvccSnapshot[applied={T|T < 7299062575647043584}]
I20260620 23:21:39.175570 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:39.169923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35947 (local address 127.0.97.2:44889)
0620 23:21:39.170156 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:39.170161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:39.170177 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:21:39.170236 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:39.170242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:39.170321 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:39.170415 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:21:39.170956 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.171786 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.173810 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.174439 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.174567 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:39.174997 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:21:39.175086 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:21:39.175152 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:39.175294 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 23:21:39.175349 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260620 23:21:39.175734 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:39.169507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44889 (local address 127.0.97.3:35947)
0620 23:21:39.169906 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:39.169927 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:39.170031 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:39.170445 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:39.170450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:39.170463 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:21:39.170730 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:39.170742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.171921 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.171927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:39.173652 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:39.173673 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:39.174578 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:39.174584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:39.174794 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:39.174856 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:21:39.175178 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:21:39.175188 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:21:39.175272 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:21:39.175416 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
W20260620 23:21:39.177229 2671 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:21:40.777076 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:40.777261 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:40.780947 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f: Sending DeleteTablet for 3 replicas of tablet cf95e386f7f5418ca012d323830cf37e
I20260620 23:21:40.781692 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet cf95e386f7f5418ca012d323830cf37e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57830
I20260620 23:21:40.781701 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet cf95e386f7f5418ca012d323830cf37e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45168
I20260620 23:21:40.781718 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet cf95e386f7f5418ca012d323830cf37e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44802
I20260620 23:21:40.791780 2734 tablet_replica.cc:330] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98: stopping tablet replica
I20260620 23:21:40.792064 2734 raft_consensus.cc:2229] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:40.792320 2734 raft_consensus.cc:2258] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:40.792876 2733 tablet_replica.cc:330] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59: stopping tablet replica
I20260620 23:21:40.793087 2733 raft_consensus.cc:2229] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:40.793327 2733 raft_consensus.cc:2258] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:40.793579 2734 ts_tablet_manager.cc:1905] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:40.794203 2735 tablet_replica.cc:330] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938: stopping tablet replica
I20260620 23:21:40.794425 2735 raft_consensus.cc:2229] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:21:40.794780 2733 ts_tablet_manager.cc:1905] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:40.794989 2735 raft_consensus.cc:2258] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:40.795882 2735 ts_tablet_manager.cc:1905] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:40.800138 2734 ts_tablet_manager.cc:1918] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:21:40.800312 2734 log.cc:1196] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/cf95e386f7f5418ca012d323830cf37e
I20260620 23:21:40.800846 2734 ts_tablet_manager.cc:1939] T cf95e386f7f5418ca012d323830cf37e P 722fdf275bca4067b47b0fcd696c0c98: Deleting consensus metadata
I20260620 23:21:40.802390 2733 ts_tablet_manager.cc:1918] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:21:40.802600 2733 log.cc:1196] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/cf95e386f7f5418ca012d323830cf37e
I20260620 23:21:40.803148 2733 ts_tablet_manager.cc:1939] T cf95e386f7f5418ca012d323830cf37e P 733dd09c7b284d9fa3cb994a1c9b6e59: Deleting consensus metadata
I20260620 23:21:40.803313 2735 ts_tablet_manager.cc:1918] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:21:40.803510 2735 log.cc:1196] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/cf95e386f7f5418ca012d323830cf37e
I20260620 23:21:40.803887 2124 catalog_manager.cc:4841] TS 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969): tablet cf95e386f7f5418ca012d323830cf37e (table test-table [id=c82bb018e7b046cca2d29c8184ecf3a6]) successfully deleted
I20260620 23:21:40.803978 2735 ts_tablet_manager.cc:1939] T cf95e386f7f5418ca012d323830cf37e P 0e540c1abb104d4a8c7ff3931c4f5938: Deleting consensus metadata
I20260620 23:21:40.804203 2125 catalog_manager.cc:4841] TS 733dd09c7b284d9fa3cb994a1c9b6e59 (127.0.97.3:44083): tablet cf95e386f7f5418ca012d323830cf37e (table test-table [id=c82bb018e7b046cca2d29c8184ecf3a6]) successfully deleted
I20260620 23:21:40.805039 2124 catalog_manager.cc:4841] TS 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889): tablet cf95e386f7f5418ca012d323830cf37e (table test-table [id=c82bb018e7b046cca2d29c8184ecf3a6]) successfully deleted
Jun 20 23:21:40 dist-test-slave-5kzz krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:21:40 dist-test-slave-5kzz krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:40.835035 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:40.821973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.0.97.62:34167)
0620 23:21:40.822503 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:40.822507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:40.822527 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:21:40.822666 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:40.822669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:40.822797 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:40.822929 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:40.824176 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:40.824941 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:40.826149 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:40.826511 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:40.826602 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:40.830296 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:40.830301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:40.830304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:40.830331 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:40.832882 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:40.833439 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:40.833443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:40.833445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:40.833564 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:40.833855 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:40.833858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:40.833860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:40.834105 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:40.834266 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:40.834567 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 23:21:40.834792 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":267,"threads_started":1}
I20260620 23:21:40.838349 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:21:40.838961 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:21:40.844899 2415 tablet_service.cc:1467] Processing CreateTablet for tablet db885c7d14f04ebd8155ba4a5b0e0723 (DEFAULT_TABLE table=test-table [id=a46ffac8095b4ec98ed9f445a7fbe141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:40.844904 2276 tablet_service.cc:1467] Processing CreateTablet for tablet db885c7d14f04ebd8155ba4a5b0e0723 (DEFAULT_TABLE table=test-table [id=a46ffac8095b4ec98ed9f445a7fbe141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:40.845021 2554 tablet_service.cc:1467] Processing CreateTablet for tablet db885c7d14f04ebd8155ba4a5b0e0723 (DEFAULT_TABLE table=test-table [id=a46ffac8095b4ec98ed9f445a7fbe141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:40.845359 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db885c7d14f04ebd8155ba4a5b0e0723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:40.845357 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db885c7d14f04ebd8155ba4a5b0e0723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:40.845410 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db885c7d14f04ebd8155ba4a5b0e0723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:40.851114 2744 tablet_bootstrap.cc:492] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98: Bootstrap starting.
I20260620 23:21:40.852360 2745 tablet_bootstrap.cc:492] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938: Bootstrap starting.
I20260620 23:21:40.852672 2746 tablet_bootstrap.cc:492] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59: Bootstrap starting.
I20260620 23:21:40.853248 2744 tablet_bootstrap.cc:654] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:40.853979 2745 tablet_bootstrap.cc:654] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:40.854434 2746 tablet_bootstrap.cc:654] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:40.855481 2744 tablet_bootstrap.cc:492] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98: No bootstrap required, opened a new log
I20260620 23:21:40.855614 2744 ts_tablet_manager.cc:1397] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 23:21:40.856546 2746 tablet_bootstrap.cc:492] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59: No bootstrap required, opened a new log
I20260620 23:21:40.856407 2744 raft_consensus.cc:348] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:40.856649 2744 raft_consensus.cc:374] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:40.856684 2746 ts_tablet_manager.cc:1397] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:21:40.856742 2744 raft_consensus.cc:729] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722fdf275bca4067b47b0fcd696c0c98, State: Initialized, Role: FOLLOWER
I20260620 23:21:40.856776 2745 tablet_bootstrap.cc:492] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938: No bootstrap required, opened a new log
I20260620 23:21:40.856894 2745 ts_tablet_manager.cc:1397] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 23:21:40.856978 2744 consensus_queue.cc:260] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:40.857786 2746 raft_consensus.cc:348] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:40.858018 2746 raft_consensus.cc:374] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:40.857832 2745 raft_consensus.cc:348] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:40.858107 2746 raft_consensus.cc:729] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733dd09c7b284d9fa3cb994a1c9b6e59, State: Initialized, Role: FOLLOWER
I20260620 23:21:40.858134 2745 raft_consensus.cc:374] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:40.858222 2745 raft_consensus.cc:729] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e540c1abb104d4a8c7ff3931c4f5938, State: Initialized, Role: FOLLOWER
I20260620 23:21:40.858317 2746 consensus_queue.cc:260] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:40.858455 2745 consensus_queue.cc:260] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:40.858870 2746 ts_tablet_manager.cc:1428] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:21:40.859141 2745 ts_tablet_manager.cc:1428] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:21:40.864010 2744 ts_tablet_manager.cc:1428] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98: Time spent starting tablet: real 0.008s user 0.000s sys 0.008s
I20260620 23:21:40.950019 2713 raft_consensus.cc:482] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:21:40.950271 2713 raft_consensus.cc:504] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:40.950912 2713 leader_election.cc:288] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969), 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889)
I20260620 23:21:40.951485 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db885c7d14f04ebd8155ba4a5b0e0723" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722fdf275bca4067b47b0fcd696c0c98" is_pre_election: true
I20260620 23:21:40.951537 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db885c7d14f04ebd8155ba4a5b0e0723" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" is_pre_election: true
I20260620 23:21:40.951710 2295 raft_consensus.cc:2454] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 0.
I20260620 23:21:40.951845 2438 raft_consensus.cc:2454] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 0.
I20260620 23:21:40.952157 2506 leader_election.cc:302] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722fdf275bca4067b47b0fcd696c0c98, 733dd09c7b284d9fa3cb994a1c9b6e59; no voters:
I20260620 23:21:40.952399 2713 raft_consensus.cc:2781] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:21:40.952512 2713 raft_consensus.cc:482] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:21:40.952600 2713 raft_consensus.cc:3037] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:40.954546 2713 raft_consensus.cc:504] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:40.955147 2713 leader_election.cc:288] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 election: Requested vote from peers 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969), 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889)
I20260620 23:21:40.955587 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db885c7d14f04ebd8155ba4a5b0e0723" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722fdf275bca4067b47b0fcd696c0c98"
I20260620 23:21:40.955638 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db885c7d14f04ebd8155ba4a5b0e0723" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e540c1abb104d4a8c7ff3931c4f5938"
I20260620 23:21:40.955798 2295 raft_consensus.cc:3037] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:40.955814 2438 raft_consensus.cc:3037] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:40.958612 2438 raft_consensus.cc:2454] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 1.
I20260620 23:21:40.958612 2295 raft_consensus.cc:2454] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 1.
I20260620 23:21:40.958992 2506 leader_election.cc:302] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722fdf275bca4067b47b0fcd696c0c98, 733dd09c7b284d9fa3cb994a1c9b6e59; no voters:
I20260620 23:21:40.959221 2713 raft_consensus.cc:2781] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:40.959368 2713 raft_consensus.cc:686] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 LEADER]: Becoming Leader. State: Replica: 733dd09c7b284d9fa3cb994a1c9b6e59, State: Running, Role: LEADER
I20260620 23:21:40.959643 2713 consensus_queue.cc:237] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:40.962034 2139 catalog_manager.cc:5495] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 733dd09c7b284d9fa3cb994a1c9b6e59 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } health_report { overall_health: HEALTHY } } }
I20260620 23:21:40.985075 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:40.980709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.0.97.3:44083)
0620 23:21:40.981185 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:40.981189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:40.981206 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:21:40.981251 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:40.981254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:40.981333 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:40.981413 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:21:40.981715 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:40.982454 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:40.983658 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:40.984059 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:40.984151 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:40.984497 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:21:40.984595 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:21:40.984659 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:40.984778 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 23:21:40.984838 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":274,"threads_started":1}
I20260620 23:21:40.992620 2438 raft_consensus.cc:1261] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Refusing update from remote peer 733dd09c7b284d9fa3cb994a1c9b6e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:40.992677 2295 raft_consensus.cc:1261] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Refusing update from remote peer 733dd09c7b284d9fa3cb994a1c9b6e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:40.993170 2713 consensus_queue.cc:1035] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:40.993445 2718 consensus_queue.cc:1035] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:41.032665 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:41.032970 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:41.035748 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f: Sending DeleteTablet for 3 replicas of tablet db885c7d14f04ebd8155ba4a5b0e0723
I20260620 23:21:41.036428 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet db885c7d14f04ebd8155ba4a5b0e0723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44802
I20260620 23:21:41.036485 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet db885c7d14f04ebd8155ba4a5b0e0723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57830
I20260620 23:21:41.036518 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet db885c7d14f04ebd8155ba4a5b0e0723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45168
I20260620 23:21:41.036701 2733 tablet_replica.cc:330] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59: stopping tablet replica
I20260620 23:21:41.036789 2735 tablet_replica.cc:330] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938: stopping tablet replica
I20260620 23:21:41.036818 2734 tablet_replica.cc:330] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98: stopping tablet replica
I20260620 23:21:41.036899 2733 raft_consensus.cc:2229] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:21:41.036988 2735 raft_consensus.cc:2229] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:41.037020 2734 raft_consensus.cc:2229] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:41.037147 2735 raft_consensus.cc:2258] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:41.037180 2733 raft_consensus.cc:2258] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:41.037196 2734 raft_consensus.cc:2258] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:41.038048 2733 ts_tablet_manager.cc:1905] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:41.038178 2735 ts_tablet_manager.cc:1905] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:41.038419 2734 ts_tablet_manager.cc:1905] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:41.038578 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:21:41.039146 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:21:41.043838 2733 ts_tablet_manager.cc:1918] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:21:41.043993 2733 log.cc:1196] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/db885c7d14f04ebd8155ba4a5b0e0723
I20260620 23:21:41.044399 2733 ts_tablet_manager.cc:1939] T db885c7d14f04ebd8155ba4a5b0e0723 P 733dd09c7b284d9fa3cb994a1c9b6e59: Deleting consensus metadata
I20260620 23:21:41.044904 2735 ts_tablet_manager.cc:1918] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:21:41.045109 2735 log.cc:1196] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/db885c7d14f04ebd8155ba4a5b0e0723
I20260620 23:21:41.045164 2734 ts_tablet_manager.cc:1918] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:21:41.045305 2734 log.cc:1196] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/db885c7d14f04ebd8155ba4a5b0e0723
I20260620 23:21:41.045614 2735 ts_tablet_manager.cc:1939] T db885c7d14f04ebd8155ba4a5b0e0723 P 0e540c1abb104d4a8c7ff3931c4f5938: Deleting consensus metadata
I20260620 23:21:41.045684 2734 ts_tablet_manager.cc:1939] T db885c7d14f04ebd8155ba4a5b0e0723 P 722fdf275bca4067b47b0fcd696c0c98: Deleting consensus metadata
I20260620 23:21:41.045902 2125 catalog_manager.cc:4841] TS 733dd09c7b284d9fa3cb994a1c9b6e59 (127.0.97.3:44083): tablet db885c7d14f04ebd8155ba4a5b0e0723 (table test-table [id=a46ffac8095b4ec98ed9f445a7fbe141]) successfully deleted
I20260620 23:21:41.046548 2124 catalog_manager.cc:4841] TS 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889): tablet db885c7d14f04ebd8155ba4a5b0e0723 (table test-table [id=a46ffac8095b4ec98ed9f445a7fbe141]) successfully deleted
I20260620 23:21:41.046819 2124 catalog_manager.cc:4841] TS 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969): tablet db885c7d14f04ebd8155ba4a5b0e0723 (table test-table [id=a46ffac8095b4ec98ed9f445a7fbe141]) successfully deleted
I20260620 23:21:41.048532 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 01567605a2604128be67861b0f648438 (DEFAULT_TABLE table=test-table [id=63356f6ae54a4f9d83b04c4dd23b5553]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:41.048741 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 01567605a2604128be67861b0f648438 (DEFAULT_TABLE table=test-table [id=63356f6ae54a4f9d83b04c4dd23b5553]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:41.048759 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 01567605a2604128be67861b0f648438 (DEFAULT_TABLE table=test-table [id=63356f6ae54a4f9d83b04c4dd23b5553]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:41.048913 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01567605a2604128be67861b0f648438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:41.049110 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01567605a2604128be67861b0f648438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:41.049120 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01567605a2604128be67861b0f648438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:41.053838 2744 tablet_bootstrap.cc:492] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98: Bootstrap starting.
I20260620 23:21:41.055186 2744 tablet_bootstrap.cc:654] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:41.055320 2745 tablet_bootstrap.cc:492] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938: Bootstrap starting.
I20260620 23:21:41.056870 2745 tablet_bootstrap.cc:654] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:41.057067 2744 tablet_bootstrap.cc:492] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98: No bootstrap required, opened a new log
I20260620 23:21:41.057224 2744 ts_tablet_manager.cc:1397] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 23:21:41.058374 2746 tablet_bootstrap.cc:492] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59: Bootstrap starting.
I20260620 23:21:41.058286 2744 raft_consensus.cc:348] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:41.058518 2744 raft_consensus.cc:374] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:41.058600 2744 raft_consensus.cc:729] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722fdf275bca4067b47b0fcd696c0c98, State: Initialized, Role: FOLLOWER
I20260620 23:21:41.058846 2744 consensus_queue.cc:260] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:41.059273 2745 tablet_bootstrap.cc:492] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938: No bootstrap required, opened a new log
I20260620 23:21:41.059373 2744 ts_tablet_manager.cc:1428] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:21:41.059414 2745 ts_tablet_manager.cc:1397] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:21:41.059989 2746 tablet_bootstrap.cc:654] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:41.060334 2745 raft_consensus.cc:348] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:41.060597 2745 raft_consensus.cc:374] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:41.060679 2745 raft_consensus.cc:729] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e540c1abb104d4a8c7ff3931c4f5938, State: Initialized, Role: FOLLOWER
I20260620 23:21:41.060954 2745 consensus_queue.cc:260] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:41.061610 2745 ts_tablet_manager.cc:1428] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:21:41.062882 2746 tablet_bootstrap.cc:492] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59: No bootstrap required, opened a new log
I20260620 23:21:41.063042 2746 ts_tablet_manager.cc:1397] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 23:21:41.063643 2746 raft_consensus.cc:348] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:41.063930 2746 raft_consensus.cc:374] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:41.064018 2746 raft_consensus.cc:729] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733dd09c7b284d9fa3cb994a1c9b6e59, State: Initialized, Role: FOLLOWER
I20260620 23:21:41.064247 2746 consensus_queue.cc:260] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:41.064864 2746 ts_tablet_manager.cc:1428] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:21:41.243058 2713 raft_consensus.cc:482] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:21:41.243206 2713 raft_consensus.cc:504] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:41.243794 2713 leader_election.cc:288] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969), 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889)
I20260620 23:21:41.244366 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01567605a2604128be67861b0f648438" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722fdf275bca4067b47b0fcd696c0c98" is_pre_election: true
I20260620 23:21:41.244386 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01567605a2604128be67861b0f648438" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" is_pre_election: true
I20260620 23:21:41.244611 2296 raft_consensus.cc:2454] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 0.
I20260620 23:21:41.244649 2438 raft_consensus.cc:2454] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 0.
I20260620 23:21:41.245074 2506 leader_election.cc:302] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722fdf275bca4067b47b0fcd696c0c98, 733dd09c7b284d9fa3cb994a1c9b6e59; no voters:
I20260620 23:21:41.245308 2713 raft_consensus.cc:2781] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:21:41.245416 2713 raft_consensus.cc:482] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:21:41.245503 2713 raft_consensus.cc:3037] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:41.247583 2713 raft_consensus.cc:504] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:41.248150 2713 leader_election.cc:288] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 election: Requested vote from peers 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969), 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889)
I20260620 23:21:41.248613 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01567605a2604128be67861b0f648438" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722fdf275bca4067b47b0fcd696c0c98"
I20260620 23:21:41.248706 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01567605a2604128be67861b0f648438" candidate_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e540c1abb104d4a8c7ff3931c4f5938"
I20260620 23:21:41.248800 2296 raft_consensus.cc:3037] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:41.248896 2438 raft_consensus.cc:3037] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:41.252127 2296 raft_consensus.cc:2454] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 1.
I20260620 23:21:41.252282 2438 raft_consensus.cc:2454] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733dd09c7b284d9fa3cb994a1c9b6e59 in term 1.
I20260620 23:21:41.252533 2506 leader_election.cc:302] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722fdf275bca4067b47b0fcd696c0c98, 733dd09c7b284d9fa3cb994a1c9b6e59; no voters:
I20260620 23:21:41.252803 2713 raft_consensus.cc:2781] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:41.252959 2713 raft_consensus.cc:686] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 LEADER]: Becoming Leader. State: Replica: 733dd09c7b284d9fa3cb994a1c9b6e59, State: Running, Role: LEADER
I20260620 23:21:41.253225 2713 consensus_queue.cc:237] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } }
I20260620 23:21:41.255931 2139 catalog_manager.cc:5495] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 733dd09c7b284d9fa3cb994a1c9b6e59 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733dd09c7b284d9fa3cb994a1c9b6e59" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44083 } health_report { overall_health: HEALTHY } } }
I20260620 23:21:41.304370 2438 raft_consensus.cc:1261] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Refusing update from remote peer 733dd09c7b284d9fa3cb994a1c9b6e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:41.304606 2295 raft_consensus.cc:1261] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Refusing update from remote peer 733dd09c7b284d9fa3cb994a1c9b6e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:41.304931 2713 consensus_queue.cc:1035] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e540c1abb104d4a8c7ff3931c4f5938" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:41.305197 2677 consensus_queue.cc:1035] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "722fdf275bca4067b47b0fcd696c0c98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:21:41.322216 2711 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:21:42.898032 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:42.898232 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:42.901849 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a69992dc0903466da1e7f57f7c747a8f: Sending DeleteTablet for 3 replicas of tablet 01567605a2604128be67861b0f648438
I20260620 23:21:42.902617 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet 01567605a2604128be67861b0f648438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57830
I20260620 23:21:42.902642 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet 01567605a2604128be67861b0f648438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44802
I20260620 23:21:42.902644 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet 01567605a2604128be67861b0f648438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45168
I20260620 23:21:42.903199 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 2210
I20260620 23:21:42.915023 2785 tablet_replica.cc:330] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938: stopping tablet replica
I20260620 23:21:42.915521 2783 tablet_replica.cc:330] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98: stopping tablet replica
I20260620 23:21:42.915910 2783 raft_consensus.cc:2229] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:42.916184 2783 raft_consensus.cc:2258] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:42.916545 2785 raft_consensus.cc:2229] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:42.916849 2785 raft_consensus.cc:2258] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:42.917747 2783 ts_tablet_manager.cc:1905] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:42.918036 2785 ts_tablet_manager.cc:1905] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:42.920353 2784 tablet_replica.cc:330] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59: stopping tablet replica
I20260620 23:21:42.920588 2784 raft_consensus.cc:2229] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:21:42.920989 2784 raft_consensus.cc:2258] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:42.923301 2784 ts_tablet_manager.cc:1905] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:42.925132 2785 ts_tablet_manager.cc:1918] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:21:42.925283 2785 log.cc:1196] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/01567605a2604128be67861b0f648438
I20260620 23:21:42.925760 2785 ts_tablet_manager.cc:1939] T 01567605a2604128be67861b0f648438 P 0e540c1abb104d4a8c7ff3931c4f5938: Deleting consensus metadata
I20260620 23:21:42.926823 2124 catalog_manager.cc:4841] TS 0e540c1abb104d4a8c7ff3931c4f5938 (127.0.97.2:44889): tablet 01567605a2604128be67861b0f648438 (table test-table [id=63356f6ae54a4f9d83b04c4dd23b5553]) successfully deleted
I20260620 23:21:42.927404 2783 ts_tablet_manager.cc:1918] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:21:42.927558 2783 log.cc:1196] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/01567605a2604128be67861b0f648438
I20260620 23:21:42.928018 2783 ts_tablet_manager.cc:1939] T 01567605a2604128be67861b0f648438 P 722fdf275bca4067b47b0fcd696c0c98: Deleting consensus metadata
I20260620 23:21:42.928858 2784 ts_tablet_manager.cc:1918] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:21:42.928980 2784 log.cc:1196] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/01567605a2604128be67861b0f648438
I20260620 23:21:42.929018 2124 catalog_manager.cc:4841] TS 722fdf275bca4067b47b0fcd696c0c98 (127.0.97.1:35969): tablet 01567605a2604128be67861b0f648438 (table test-table [id=63356f6ae54a4f9d83b04c4dd23b5553]) successfully deleted
I20260620 23:21:42.929404 2784 ts_tablet_manager.cc:1939] T 01567605a2604128be67861b0f648438 P 733dd09c7b284d9fa3cb994a1c9b6e59: Deleting consensus metadata
I20260620 23:21:42.930388 2125 catalog_manager.cc:4841] TS 733dd09c7b284d9fa3cb994a1c9b6e59 (127.0.97.3:44083): tablet 01567605a2604128be67861b0f648438 (table test-table [id=63356f6ae54a4f9d83b04c4dd23b5553]) successfully deleted
Jun 20 23:21:42 dist-test-slave-5kzz krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:21:42.945715 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:42.905154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.0.97.1:35969)
0620 23:21:42.929123 (+ 23969us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:42.929128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:42.929150 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:21:42.929201 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:42.929206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:42.929275 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:42.929363 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:42.930941 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:42.931680 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:42.932774 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:42.933154 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:42.933239 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:42.940959 (+ 7720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:42.940972 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:42.940990 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:42.941026 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:42.943518 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:42.944087 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:42.944094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:42.944097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:42.944213 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:42.944501 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:42.944504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:42.944506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:42.944805 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:42.945021 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:42.945229 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 23:21:42.945474 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23821,"thread_start_us":24450,"threads_started":1}
W20260620 23:21:42.946346 2341 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58632
W20260620 23:21:42.946964 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 2210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:21:42.947178 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 2210
I20260620 23:21:42.965126 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 2350
Jun 20 23:21:42 dist-test-slave-5kzz krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:21:42.979673 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:42.965848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.0.97.2:44889)
0620 23:21:42.966830 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:42.966834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:42.966849 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:21:42.966902 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:42.966906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:42.966962 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:42.967058 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:42.968344 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:42.969062 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:42.970243 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:42.970615 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:42.970897 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:42.974562 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:42.974575 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:42.974589 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:42.974622 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:42.977226 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:42.977773 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:42.977779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:42.977781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:42.977896 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:42.978198 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:42.978201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:42.978202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:42.978512 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:42.978808 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:42.979054 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 23:21:42.979256 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":720,"threads_started":1}
W20260620 23:21:42.980379 2480 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55404
W20260620 23:21:42.980960 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 2350: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:21:42.981108 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 2350
I20260620 23:21:43.003302 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 2489
Jun 20 23:21:43 dist-test-slave-5kzz krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:21:43.017359 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:43.004155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.0.97.3:44083)
0620 23:21:43.004674 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:43.004678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:43.004693 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:21:43.004743 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:43.004747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:43.004805 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:43.004893 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:43.006091 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:43.006903 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:43.008039 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:43.008398 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:43.008482 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:43.012474 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:43.012487 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:43.012512 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:43.012543 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:43.015293 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:43.015833 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:43.015839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:43.015841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:43.015949 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:43.016230 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:43.016233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:43.016235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:43.016557 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:43.016743 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:43.016958 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 23:21:43.017102 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":273,"threads_started":1}
W20260620 23:21:43.017988 2619 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52382
W20260620 23:21:43.018571 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 2489: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:21:43.018757 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 2489
I20260620 23:21:43.043027 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 2107
I20260620 23:21:43.044163 2179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:43.153798 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 2107
2026-06-20T23:21:43Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8672 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:21:43 dist-test-slave-5kzz krb5kdc[2793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:21:43 dist-test-slave-5kzz krb5kdc[2793](info): set up 2 sockets
Jun 20 23:21:43 dist-test-slave-5kzz krb5kdc[2793](info): commencing operation
krb5kdc: starting...
W20260620 23:21:44.227195 388 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:21:44 dist-test-slave-5kzz krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:21:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:21:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:21:44.399647 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34201
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:44.702838 2806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:44.703094 2806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:44.703243 2806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:44.709782 2806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:21:44.709863 2806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:44.709903 2806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:44.709945 2806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:21:44.710031 2806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:21:44.719343 2806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:44.720562 2806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:44.721818 2806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:44.733280 2814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:44.733323 2811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:44.733336 2812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:44.733670 2806 server_base.cc:1034] running on GCE node
I20260620 23:21:44.734762 2806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:44.736091 2806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:44.737285 2806 hybrid_clock.cc:648] HybridClock initialized: now 1781997704737215 us; error 64 us; skew 500 ppm
I20260620 23:21:44.737638 2806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:44 dist-test-slave-5kzz krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:44.740769 2806 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:21:44.742033 2806 webserver.cc:460] Webserver started at http://127.0.97.62:33407/ using document root <none> and password file <none>
I20260620 23:21:44.742619 2806 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:44.742766 2806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:44.743024 2806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:44.745844 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "9807473182e049b18490cd438d1680f8"
format_stamp: "Formatted at 2026-06-20 23:21:44 on dist-test-slave-5kzz"
server_key: "db1ac091db11ac7b56c53a1087f5b863"
server_key_iv: "80c508638be45c80e061c5c76652ba1b"
server_key_version: "encryptionkey@0"
I20260620 23:21:44.746532 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "9807473182e049b18490cd438d1680f8"
format_stamp: "Formatted at 2026-06-20 23:21:44 on dist-test-slave-5kzz"
server_key: "db1ac091db11ac7b56c53a1087f5b863"
server_key_iv: "80c508638be45c80e061c5c76652ba1b"
server_key_version: "encryptionkey@0"
I20260620 23:21:44.751215 2806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:21:44.754271 2821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:44.755551 2806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:21:44.755681 2806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "9807473182e049b18490cd438d1680f8"
format_stamp: "Formatted at 2026-06-20 23:21:44 on dist-test-slave-5kzz"
server_key: "db1ac091db11ac7b56c53a1087f5b863"
server_key_iv: "80c508638be45c80e061c5c76652ba1b"
server_key_version: "encryptionkey@0"
I20260620 23:21:44.755832 2806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:44.778539 2806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:44.782138 2806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:44.782351 2806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:44.799149 2806 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34201
I20260620 23:21:44.799223 2872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34201 every 8 connection(s)
I20260620 23:21:44.800369 2806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:21:44.803735 2873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:44.806067 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 2806
I20260620 23:21:44.806335 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:21:44.807129 388 external_mini_cluster.cc:1428] Setting key f130eabbf13b86517cef103aaddf9249
I20260620 23:21:44.815609 2873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8: Bootstrap starting.
I20260620 23:21:44.819139 2873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:44.820384 2873 log.cc:824] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:21:44 dist-test-slave-5kzz krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:44.823398 2873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8: No bootstrap required, opened a new log
I20260620 23:21:44.830431 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:44.809199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.0.97.62:34201)
0620 23:21:44.809931 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:44.809942 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:44.809979 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:21:44.811598 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:44.811620 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:44.812009 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:44.812317 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:44.814102 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:44.815397 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:44.817114 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:44.817743 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:44.817864 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:44.822438 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:44.822448 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:44.822467 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:44.822521 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:44.826331 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:44.826969 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:44.826978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:44.826987 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:44.827183 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:44.827527 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:44.827532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:44.827535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:44.828157 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:44.828417 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:44.828731 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 23:21:44.829158 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":343,"threads_started":1}
I20260620 23:21:44.834285 2873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9807473182e049b18490cd438d1680f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34201 } }
I20260620 23:21:44.834582 2873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:44.834649 2873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9807473182e049b18490cd438d1680f8, State: Initialized, Role: FOLLOWER
I20260620 23:21:44.835178 2873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9807473182e049b18490cd438d1680f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34201 } }
I20260620 23:21:44.835362 2873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:21:44.835448 2873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:21:44.835563 2873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:44.837463 2873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9807473182e049b18490cd438d1680f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34201 } }
I20260620 23:21:44.837908 2873 leader_election.cc:302] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9807473182e049b18490cd438d1680f8; no voters:
I20260620 23:21:44.838378 2873 leader_election.cc:288] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:21:44.838516 2878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:44.839619 2878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [term 1 LEADER]: Becoming Leader. State: Replica: 9807473182e049b18490cd438d1680f8, State: Running, Role: LEADER
I20260620 23:21:44.840121 2878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9807473182e049b18490cd438d1680f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34201 } }
I20260620 23:21:44.840531 2873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:21:44.842746 2879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9807473182e049b18490cd438d1680f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9807473182e049b18490cd438d1680f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34201 } } }
I20260620 23:21:44.843094 2879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:44.842823 2880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9807473182e049b18490cd438d1680f8. Latest consensus state: current_term: 1 leader_uuid: "9807473182e049b18490cd438d1680f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9807473182e049b18490cd438d1680f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34201 } } }
I20260620 23:21:44.843202 2880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8 [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:44.845642 2885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:21:44.850896 2885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:21:44.858711 2885 catalog_manager.cc:1347] Generated new cluster ID: e6895f471c8a44cdb2295dcbc1e8e3ba
I20260620 23:21:44.858812 2885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:21:44.877488 2885 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:21:44.879532 2885 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:21:44.901588 2885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9807473182e049b18490cd438d1680f8: Generated new TSK 0
I20260620 23:21:44.902628 2885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:21:44.956424 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34201
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:45.265551 2899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:45.265856 2899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:45.266024 2899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:45.273205 2899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:45.273346 2899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:45.273531 2899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:21:45.283041 2899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:45.284256 2899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:45.285542 2899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:45.297950 2907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:45.297802 2905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:45.297996 2904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:45.298903 2899 server_base.cc:1034] running on GCE node
I20260620 23:21:45.299472 2899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:45.300329 2899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:45.301553 2899 hybrid_clock.cc:648] HybridClock initialized: now 1781997705301505 us; error 55 us; skew 500 ppm
I20260620 23:21:45.301846 2899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:45 dist-test-slave-5kzz krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:45.304800 2899 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:21:45.306169 2899 webserver.cc:460] Webserver started at http://127.0.97.1:34849/ using document root <none> and password file <none>
I20260620 23:21:45.306839 2899 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:45.306941 2899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:45.307220 2899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:45.309854 2899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "ba91265b87ce4f7899734cba5a70db6d"
format_stamp: "Formatted at 2026-06-20 23:21:45 on dist-test-slave-5kzz"
server_key: "5909dfcd30bc0e1205c7077a8f955531"
server_key_iv: "514915fdfdc74ad069e74323f7f7810d"
server_key_version: "encryptionkey@0"
I20260620 23:21:45.310575 2899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ba91265b87ce4f7899734cba5a70db6d"
format_stamp: "Formatted at 2026-06-20 23:21:45 on dist-test-slave-5kzz"
server_key: "5909dfcd30bc0e1205c7077a8f955531"
server_key_iv: "514915fdfdc74ad069e74323f7f7810d"
server_key_version: "encryptionkey@0"
I20260620 23:21:45.315385 2899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:21:45.318382 2914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:45.319624 2899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 23:21:45.319777 2899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "ba91265b87ce4f7899734cba5a70db6d"
format_stamp: "Formatted at 2026-06-20 23:21:45 on dist-test-slave-5kzz"
server_key: "5909dfcd30bc0e1205c7077a8f955531"
server_key_iv: "514915fdfdc74ad069e74323f7f7810d"
server_key_version: "encryptionkey@0"
I20260620 23:21:45.319998 2899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:45.343233 2899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:45.347214 2899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:45.347448 2899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:45.348259 2899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:45.349448 2899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:45.349517 2899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:45.349593 2899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:45.349634 2899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:45.379216 2899 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40679
I20260620 23:21:45.379287 3026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40679 every 8 connection(s)
I20260620 23:21:45.380414 2899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:21:45.382985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 2899
I20260620 23:21:45.383216 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:21:45.383816 388 external_mini_cluster.cc:1428] Setting key 7323f5e71a9624382fed2d50a5bf7f1b
Jun 20 23:21:45 dist-test-slave-5kzz krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:21:45.401284 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:45.382865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58167 (local address 127.0.97.62:34201)
0620 23:21:45.383356 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:45.383363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:45.384403 (+ 1040us) server_negotiation.cc:407] Connection header received
0620 23:21:45.386377 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:45.386385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:45.386467 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:45.386582 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:45.387970 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:45.389175 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:45.391148 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:45.391517 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:45.391609 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:45.395632 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:45.395636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:45.395639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:45.395667 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:45.398200 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:45.398973 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:45.398984 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:45.398987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:45.399096 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:45.399517 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:45.399521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:45.399524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:45.399782 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:45.399927 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:45.400687 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 23:21:45.400924 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":257,"threads_started":1}
I20260620 23:21:45.402220 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:45.383436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34201 (local address 127.0.97.1:58167)
0620 23:21:45.384242 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:45.384293 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:45.386127 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:45.386828 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:45.386836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:45.387292 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:45.387805 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:45.387825 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:45.389318 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:45.389322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:45.391008 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:45.391024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:45.391723 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:45.391733 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:45.392036 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:45.392792 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:45.392823 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:45.395411 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:45.398339 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:45.398349 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:45.398364 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:45.398834 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:45.399220 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:45.399225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:45.399229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:45.399403 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:45.399912 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:45.399920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:45.400272 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 23:21:45.400603 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 23:21:45.401084 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":344,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:21:45.403782 3027 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34201
I20260620 23:21:45.404088 3027 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:45.404762 3027 heartbeater.cc:511] Master 127.0.97.62:34201 requested a full tablet report, sending...
I20260620 23:21:45.406746 2838 ts_manager.cc:194] Registered new tserver with Master: ba91265b87ce4f7899734cba5a70db6d (127.0.97.1:40679)
I20260620 23:21:45.408929 2838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:21:45.423300 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34201
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:45.732549 3033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:45.732802 3033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:45.732975 3033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:45.739912 3033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:45.740044 3033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:45.740219 3033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:21:45.749758 3033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:45.751004 3033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:45.752233 3033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:45.764680 3038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:45.764724 3041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:45.764752 3039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:45.765770 3033 server_base.cc:1034] running on GCE node
I20260620 23:21:45.766378 3033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:45.767352 3033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:45.768565 3033 hybrid_clock.cc:648] HybridClock initialized: now 1781997705768506 us; error 47 us; skew 500 ppm
I20260620 23:21:45.768857 3033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:45 dist-test-slave-5kzz krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:45.771615 3033 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:21:45.772851 3033 webserver.cc:460] Webserver started at http://127.0.97.2:38295/ using document root <none> and password file <none>
I20260620 23:21:45.773509 3033 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:45.773653 3033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:45.773952 3033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:45.776697 3033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "64e88330c403489994fc7abb2928b28e"
format_stamp: "Formatted at 2026-06-20 23:21:45 on dist-test-slave-5kzz"
server_key: "20e20c08c891eb7661f30472441ed38c"
server_key_iv: "19e4e54cd0d8ff95d47fe064bc61a575"
server_key_version: "encryptionkey@0"
I20260620 23:21:45.777463 3033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "64e88330c403489994fc7abb2928b28e"
format_stamp: "Formatted at 2026-06-20 23:21:45 on dist-test-slave-5kzz"
server_key: "20e20c08c891eb7661f30472441ed38c"
server_key_iv: "19e4e54cd0d8ff95d47fe064bc61a575"
server_key_version: "encryptionkey@0"
I20260620 23:21:45.782130 3033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:21:45.785305 3048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:45.786573 3033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:21:45.786746 3033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "64e88330c403489994fc7abb2928b28e"
format_stamp: "Formatted at 2026-06-20 23:21:45 on dist-test-slave-5kzz"
server_key: "20e20c08c891eb7661f30472441ed38c"
server_key_iv: "19e4e54cd0d8ff95d47fe064bc61a575"
server_key_version: "encryptionkey@0"
I20260620 23:21:45.786907 3033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:45.837311 3033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:45.841181 3033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:45.841501 3033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:45.842432 3033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:45.843699 3033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:45.843768 3033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:45.843941 3033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:45.843993 3033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:45.875504 3033 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37655
I20260620 23:21:45.875576 3160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37655 every 8 connection(s)
I20260620 23:21:45.876689 3033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:21:45.879042 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 3033
I20260620 23:21:45.879238 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:21:45.879673 388 external_mini_cluster.cc:1428] Setting key 0ac82622e2bbc15c4bd92e586e34f9a6
Jun 20 23:21:45 dist-test-slave-5kzz krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:21:45.899302 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:45.879066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44539 (local address 127.0.97.62:34201)
0620 23:21:45.879261 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:45.879268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:45.880610 (+ 1342us) server_negotiation.cc:407] Connection header received
0620 23:21:45.882982 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:45.882988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:45.883113 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:45.883251 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:45.885356 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:45.886055 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:45.887987 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:45.888750 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:45.888892 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:45.892604 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:45.892613 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:45.892621 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:45.892662 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:45.896412 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:45.897092 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:45.897099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:45.897103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:45.897256 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:45.897640 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:45.897646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:45.897650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:45.898005 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:45.898171 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:45.898804 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 23:21:45.899082 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:21:45.900216 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:45.879625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34201 (local address 127.0.97.2:44539)
0620 23:21:45.880418 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:45.880459 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:45.882614 (+ 2155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:45.883587 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:45.883601 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:45.884387 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:45.885170 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:45.885198 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:45.886205 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:45.886212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:45.887856 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:45.887867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:45.888946 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:45.888950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:45.889137 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:45.889806 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:45.889831 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:45.892404 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:45.896548 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:45.896554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:45.896566 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:45.896955 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:45.897364 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:45.897367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:45.897369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:45.897524 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:45.898132 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:45.898137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:45.898417 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 23:21:45.898620 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 23:21:45.899101 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":339,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 23:21:45.901682 3161 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34201
I20260620 23:21:45.902005 3161 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:45.902889 3161 heartbeater.cc:511] Master 127.0.97.62:34201 requested a full tablet report, sending...
I20260620 23:21:45.904392 2838 ts_manager.cc:194] Registered new tserver with Master: 64e88330c403489994fc7abb2928b28e (127.0.97.2:37655)
I20260620 23:21:45.905161 2838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:21:45.921314 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34201
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:46.226121 3166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:46.226428 3166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:46.226594 3166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:46.233736 3166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:46.233881 3166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:46.234068 3166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:21:46.243788 3166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:46.245092 3166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:46.246385 3166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:46.258862 3171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:46.258920 3174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:46.258950 3172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:46.260180 3166 server_base.cc:1034] running on GCE node
I20260620 23:21:46.260748 3166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:46.261669 3166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:46.262877 3166 hybrid_clock.cc:648] HybridClock initialized: now 1781997706262842 us; error 49 us; skew 500 ppm
I20260620 23:21:46.263182 3166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:46 dist-test-slave-5kzz krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:46.266094 3166 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:21:46.267498 3166 webserver.cc:460] Webserver started at http://127.0.97.3:41371/ using document root <none> and password file <none>
I20260620 23:21:46.268144 3166 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:46.268280 3166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:46.268574 3166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:46.271425 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "a3753245de9c4f5c85142f4df65fb0e9"
format_stamp: "Formatted at 2026-06-20 23:21:46 on dist-test-slave-5kzz"
server_key: "ee90eb8dfec6395cc4258458b23bc344"
server_key_iv: "4e110035a6f519160c0833bac7dcb900"
server_key_version: "encryptionkey@0"
I20260620 23:21:46.272173 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a3753245de9c4f5c85142f4df65fb0e9"
format_stamp: "Formatted at 2026-06-20 23:21:46 on dist-test-slave-5kzz"
server_key: "ee90eb8dfec6395cc4258458b23bc344"
server_key_iv: "4e110035a6f519160c0833bac7dcb900"
server_key_version: "encryptionkey@0"
I20260620 23:21:46.277019 3166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:21:46.280334 3181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:46.281515 3166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:21:46.281692 3166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "a3753245de9c4f5c85142f4df65fb0e9"
format_stamp: "Formatted at 2026-06-20 23:21:46 on dist-test-slave-5kzz"
server_key: "ee90eb8dfec6395cc4258458b23bc344"
server_key_iv: "4e110035a6f519160c0833bac7dcb900"
server_key_version: "encryptionkey@0"
I20260620 23:21:46.281884 3166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:46.297966 3166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:46.301699 3166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:46.301997 3166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:46.302925 3166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:46.304152 3166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:46.304252 3166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:46.304392 3166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:46.304461 3166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:46.335726 3166 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45939
I20260620 23:21:46.335762 3293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45939 every 8 connection(s)
I20260620 23:21:46.336925 3166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:21:46.347266 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 3166
I20260620 23:21:46.347452 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:21:46.347859 388 external_mini_cluster.cc:1428] Setting key c4bac1a7d4ec1376ee0fae729811e96e
Jun 20 23:21:46 dist-test-slave-5kzz krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:46.356046 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:46.339295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53885 (local address 127.0.97.62:34201)
0620 23:21:46.339457 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:46.339462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:46.340599 (+ 1137us) server_negotiation.cc:407] Connection header received
0620 23:21:46.342495 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:46.342500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:46.342606 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:46.342740 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:46.344200 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:46.344927 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:46.346474 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:46.346941 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:46.347072 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:46.350854 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:46.350861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:46.350865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:46.350907 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:46.353205 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:46.353875 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:46.353882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:46.353885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:46.354006 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:46.354406 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:46.354413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:46.354416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:46.354727 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:46.354860 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:46.355507 (+ 647us) server_negotiation.cc:299] Negotiation successful
0620 23:21:46.355707 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:21:46.356921 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:46.339732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34201 (local address 127.0.97.3:53885)
0620 23:21:46.340451 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:46.340487 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:46.342214 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:46.342976 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:46.342985 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:46.343489 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:46.344029 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:46.344054 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:46.345095 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:46.345099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:46.346329 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:46.346340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:46.347164 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:46.347172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:46.347450 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:46.348150 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:46.348171 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:46.350609 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:46.353338 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:46.353347 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:46.353362 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:46.353759 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:46.354117 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:46.354123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:46.354126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:46.354300 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:46.354832 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:46.354840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:46.355157 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 23:21:46.355422 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 23:21:46.355906 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":272,"threads_started":1}
I20260620 23:21:46.358117 3294 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34201
I20260620 23:21:46.358419 3294 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:46.359172 3294 heartbeater.cc:511] Master 127.0.97.62:34201 requested a full tablet report, sending...
I20260620 23:21:46.360497 2838 ts_manager.cc:194] Registered new tserver with Master: a3753245de9c4f5c85142f4df65fb0e9 (127.0.97.3:45939)
I20260620 23:21:46.361282 2838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53885
I20260620 23:21:46.362960 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:21:46.390133 3029 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:21:46.388107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.0.97.62:34201)
0620 23:21:46.388292 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:46.388297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:46.388719 (+ 422us) server_negotiation.cc:407] Connection header received
0620 23:21:46.388854 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:46.388857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:46.388914 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:46.389004 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:46.389960 (+ 956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47190: BlockingRecv error: recv got EOF from 127.0.0.1:47190 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:21:46.393203 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 2899
W20260620 23:21:46.395833 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 2899: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:40679: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:21:46.396005 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 2899
I20260620 23:21:46.411015 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3033
W20260620 23:21:46.413305 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3033: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:37655: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:21:46.413455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 3033
I20260620 23:21:46.427944 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3166
W20260620 23:21:46.430294 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3166: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:45939: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:21:46.430389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 3166
I20260620 23:21:46.445268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 2806
I20260620 23:21:46.446326 2868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:46.543907 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 2806
2026-06-20T23:21:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3387 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:21:46 dist-test-slave-5kzz krb5kdc[3311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:21:46 dist-test-slave-5kzz krb5kdc[3311](info): set up 2 sockets
Jun 20 23:21:46 dist-test-slave-5kzz krb5kdc[3311](info): commencing operation
krb5kdc: starting...
W20260620 23:21:47.618933 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:21:47 dist-test-slave-5kzz krb5kdc[3311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:21:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:21:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:21:47.796126 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41337
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:48.114305 3324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:48.114675 3324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:48.114866 3324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:48.122233 3324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:21:48.122400 3324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:48.122515 3324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:48.122628 3324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:21:48.122757 3324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:21:48.132809 3324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:48.134162 3324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:48.135586 3324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:48.147861 3329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:48.147799 3330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:48.147830 3332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:48.148581 3324 server_base.cc:1034] running on GCE node
I20260620 23:21:48.149308 3324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:48.150750 3324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:48.151969 3324 hybrid_clock.cc:648] HybridClock initialized: now 1781997708151935 us; error 41 us; skew 500 ppm
I20260620 23:21:48.152297 3324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:48 dist-test-slave-5kzz krb5kdc[3311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:48.158614 3324 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:21:48.160065 3324 webserver.cc:460] Webserver started at http://127.0.97.62:44727/ using document root <none> and password file <none>
I20260620 23:21:48.160759 3324 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:48.160912 3324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:48.161209 3324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:48.164160 3324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "302e5c899a8a417599146fa90f6280ab"
format_stamp: "Formatted at 2026-06-20 23:21:48 on dist-test-slave-5kzz"
server_key: "fa5787c795ce126dec43692692da8f65"
server_key_iv: "adb9ea3c2d6bf2d9157065d79d84b104"
server_key_version: "encryptionkey@0"
I20260620 23:21:48.164908 3324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "302e5c899a8a417599146fa90f6280ab"
format_stamp: "Formatted at 2026-06-20 23:21:48 on dist-test-slave-5kzz"
server_key: "fa5787c795ce126dec43692692da8f65"
server_key_iv: "adb9ea3c2d6bf2d9157065d79d84b104"
server_key_version: "encryptionkey@0"
I20260620 23:21:48.169700 3324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:21:48.173044 3339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:48.174254 3324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:21:48.174438 3324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "302e5c899a8a417599146fa90f6280ab"
format_stamp: "Formatted at 2026-06-20 23:21:48 on dist-test-slave-5kzz"
server_key: "fa5787c795ce126dec43692692da8f65"
server_key_iv: "adb9ea3c2d6bf2d9157065d79d84b104"
server_key_version: "encryptionkey@0"
I20260620 23:21:48.174631 3324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:48.194481 3324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:48.198297 3324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:48.198587 3324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:48.216780 3324 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41337
I20260620 23:21:48.216811 3390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41337 every 8 connection(s)
I20260620 23:21:48.218093 3324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:21:48.221588 3391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:48.223994 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 3324
I20260620 23:21:48.224205 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:21:48.224822 388 external_mini_cluster.cc:1428] Setting key d07dadedbfe43847c669430cb8f0a54f
I20260620 23:21:48.233999 3391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab: Bootstrap starting.
I20260620 23:21:48.237499 3391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:48.238464 3391 log.cc:824] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:21:48 dist-test-slave-5kzz krb5kdc[3311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:48.241478 3391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab: No bootstrap required, opened a new log
I20260620 23:21:48.248190 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:48.226708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.0.97.62:41337)
0620 23:21:48.227524 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:48.227540 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:48.227589 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:21:48.229766 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:48.229794 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:48.230304 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:48.230778 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:48.232058 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:48.233959 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:48.235311 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:48.236394 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:48.236570 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:48.240729 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:48.240736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:48.240751 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:48.240787 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:48.243839 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:48.244654 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:48.244660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:48.244670 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:48.244844 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:48.245233 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:48.245236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:48.245238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:48.245733 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:48.245971 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:48.246434 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 23:21:48.246937 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":354,"threads_started":1}
I20260620 23:21:48.250224 3391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302e5c899a8a417599146fa90f6280ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41337 } }
I20260620 23:21:48.250619 3391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:48.250739 3391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302e5c899a8a417599146fa90f6280ab, State: Initialized, Role: FOLLOWER
I20260620 23:21:48.251363 3391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302e5c899a8a417599146fa90f6280ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41337 } }
I20260620 23:21:48.251579 3391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:21:48.251715 3391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:21:48.251892 3391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:48.252148 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:21:48.254037 3391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302e5c899a8a417599146fa90f6280ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41337 } }
I20260620 23:21:48.254528 3391 leader_election.cc:302] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 302e5c899a8a417599146fa90f6280ab; no voters:
I20260620 23:21:48.255117 3391 leader_election.cc:288] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:21:48.255263 3396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:48.257052 3391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:21:48.262816 3396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [term 1 LEADER]: Becoming Leader. State: Replica: 302e5c899a8a417599146fa90f6280ab, State: Running, Role: LEADER
I20260620 23:21:48.263306 3396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302e5c899a8a417599146fa90f6280ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41337 } }
I20260620 23:21:48.266283 3397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "302e5c899a8a417599146fa90f6280ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302e5c899a8a417599146fa90f6280ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41337 } } }
I20260620 23:21:48.266670 3397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:48.271595 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 302e5c899a8a417599146fa90f6280ab. Latest consensus state: current_term: 1 leader_uuid: "302e5c899a8a417599146fa90f6280ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302e5c899a8a417599146fa90f6280ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41337 } } }
I20260620 23:21:48.272481 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab [sys.catalog]: This master's current role is: LEADER
W20260620 23:21:48.279592 3411 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:21:48.279724 3411 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:21:48.279851 3412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:21:48.281970 3412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:21:48.291944 3412 catalog_manager.cc:1347] Generated new cluster ID: 974994c32fa1414fabc31fb1a3a098a8
I20260620 23:21:48.292058 3412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:21:48.298666 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:48.289551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.0.97.62:41337)
0620 23:21:48.289728 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:48.289736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:48.290409 (+ 673us) server_negotiation.cc:407] Connection header received
0620 23:21:48.290603 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:48.290608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:48.290741 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:48.290863 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:48.292053 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:48.293179 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:48.295048 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:48.295813 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:48.295947 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:48.297100 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:48.297106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:48.297111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:21:48.297819 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:48.297889 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:21:48.297912 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:48.297989 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:48.298226 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:21:48.298319 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:21:48.312261 3412 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:21:48.313565 3412 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:21:48.332599 3412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 302e5c899a8a417599146fa90f6280ab: Generated new TSK 0
I20260620 23:21:48.333634 3412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:21:48.348491 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3324
I20260620 23:21:48.349622 3386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:48.440680 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 3324
2026-06-20T23:21:48Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1891 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:21:48 dist-test-slave-5kzz krb5kdc[3425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:21:48 dist-test-slave-5kzz krb5kdc[3425](info): set up 2 sockets
Jun 20 23:21:48 dist-test-slave-5kzz krb5kdc[3425](info): commencing operation
krb5kdc: starting...
W20260620 23:21:49.519008 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:21:49 dist-test-slave-5kzz krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:21:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:21:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:21:49.707695 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33577
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:50.026208 3438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:50.026584 3438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:50.026787 3438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:50.034022 3438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:21:50.034183 3438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:50.034299 3438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:50.034399 3438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:21:50.034490 3438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:21:50.044473 3438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:50.045840 3438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:50.047224 3438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:50.059069 3443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:50.059165 3446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:50.059139 3444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:50.060107 3438 server_base.cc:1034] running on GCE node
I20260620 23:21:50.060863 3438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:50.062247 3438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:50.063480 3438 hybrid_clock.cc:648] HybridClock initialized: now 1781997710063341 us; error 147 us; skew 500 ppm
I20260620 23:21:50.063839 3438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:50 dist-test-slave-5kzz krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:50.066946 3438 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:21:50.068279 3438 webserver.cc:460] Webserver started at http://127.0.97.62:34035/ using document root <none> and password file <none>
I20260620 23:21:50.068912 3438 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:50.069005 3438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:50.069228 3438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:50.072155 3438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "b42f09276f634bb4b2b9da42f6f347ca"
format_stamp: "Formatted at 2026-06-20 23:21:50 on dist-test-slave-5kzz"
server_key: "f73c17644e619b963bd05f8d2560da1f"
server_key_iv: "36237dc90da1c9d9094ca0b70a6ce627"
server_key_version: "encryptionkey@0"
I20260620 23:21:50.072870 3438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "b42f09276f634bb4b2b9da42f6f347ca"
format_stamp: "Formatted at 2026-06-20 23:21:50 on dist-test-slave-5kzz"
server_key: "f73c17644e619b963bd05f8d2560da1f"
server_key_iv: "36237dc90da1c9d9094ca0b70a6ce627"
server_key_version: "encryptionkey@0"
I20260620 23:21:50.077890 3438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:21:50.081207 3453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:50.082528 3438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:21:50.082732 3438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "b42f09276f634bb4b2b9da42f6f347ca"
format_stamp: "Formatted at 2026-06-20 23:21:50 on dist-test-slave-5kzz"
server_key: "f73c17644e619b963bd05f8d2560da1f"
server_key_iv: "36237dc90da1c9d9094ca0b70a6ce627"
server_key_version: "encryptionkey@0"
I20260620 23:21:50.082913 3438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:50.112653 3438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:50.117059 3438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:50.117295 3438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:50.136171 3438 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33577
I20260620 23:21:50.136211 3504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33577 every 8 connection(s)
I20260620 23:21:50.137557 3438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:21:50.141208 3505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:50.146226 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 3438
I20260620 23:21:50.146445 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:21:50.147315 388 external_mini_cluster.cc:1428] Setting key dd163d4e644bb1bc11fa75a70f4af035
I20260620 23:21:50.153993 3505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca: Bootstrap starting.
I20260620 23:21:50.157588 3505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:50.158767 3505 log.cc:824] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:50.162092 3505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca: No bootstrap required, opened a new log
Jun 20 23:21:50 dist-test-slave-5kzz krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:50.169822 3505 raft_consensus.cc:348] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42f09276f634bb4b2b9da42f6f347ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33577 } }
I20260620 23:21:50.170231 3505 raft_consensus.cc:374] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:50.170352 3505 raft_consensus.cc:729] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b42f09276f634bb4b2b9da42f6f347ca, State: Initialized, Role: FOLLOWER
I20260620 23:21:50.170943 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:50.149391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45944 (local address 127.0.97.62:33577)
0620 23:21:50.150200 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:50.150212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:50.150253 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:21:50.152322 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:50.152346 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:50.152761 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:50.153099 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:50.154875 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:50.156188 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:50.157903 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:50.158570 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:50.158758 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:50.163267 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:50.163274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:50.163289 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:50.163326 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:50.166491 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:50.167304 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:50.167318 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:50.167324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:50.167495 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:50.167910 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:50.167914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:50.167916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:50.168487 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:50.168833 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:50.169195 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 23:21:50.169612 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":374,"threads_started":1}
I20260620 23:21:50.171082 3505 consensus_queue.cc:260] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42f09276f634bb4b2b9da42f6f347ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33577 } }
I20260620 23:21:50.171363 3505 raft_consensus.cc:388] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:21:50.171530 3505 raft_consensus.cc:482] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:21:50.171725 3505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:50.174185 3505 raft_consensus.cc:504] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42f09276f634bb4b2b9da42f6f347ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33577 } }
I20260620 23:21:50.174794 3505 leader_election.cc:302] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b42f09276f634bb4b2b9da42f6f347ca; no voters:
I20260620 23:21:50.175365 3505 leader_election.cc:288] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:21:50.175490 3510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:50.176201 3510 raft_consensus.cc:686] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [term 1 LEADER]: Becoming Leader. State: Replica: b42f09276f634bb4b2b9da42f6f347ca, State: Running, Role: LEADER
I20260620 23:21:50.176705 3510 consensus_queue.cc:237] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42f09276f634bb4b2b9da42f6f347ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33577 } }
I20260620 23:21:50.177464 3505 sys_catalog.cc:564] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:21:50.180073 3511 sys_catalog.cc:455] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b42f09276f634bb4b2b9da42f6f347ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42f09276f634bb4b2b9da42f6f347ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33577 } } }
I20260620 23:21:50.180496 3511 sys_catalog.cc:458] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:50.185420 3512 sys_catalog.cc:455] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b42f09276f634bb4b2b9da42f6f347ca. Latest consensus state: current_term: 1 leader_uuid: "b42f09276f634bb4b2b9da42f6f347ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42f09276f634bb4b2b9da42f6f347ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33577 } } }
I20260620 23:21:50.186340 3512 sys_catalog.cc:458] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:50.187312 3516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:21:50.192216 3516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:21:50.200685 3516 catalog_manager.cc:1347] Generated new cluster ID: 833677b8631e4217b2cffb4937b1cf01
I20260620 23:21:50.200814 3516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:21:50.212879 3516 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:21:50.214232 3516 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:21:50.228211 3516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b42f09276f634bb4b2b9da42f6f347ca: Generated new TSK 0
I20260620 23:21:50.229125 3516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:21:50.276453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33577
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:50.603756 3531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:50.604015 3531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:50.604169 3531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:50.611277 3531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:50.611392 3531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:50.611603 3531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:21:50.621266 3531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:50.622522 3531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:50.623863 3531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:50.636656 3536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:50.636693 3537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:50.636823 3539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:50.637703 3531 server_base.cc:1034] running on GCE node
I20260620 23:21:50.638293 3531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:50.639222 3531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:50.640435 3531 hybrid_clock.cc:648] HybridClock initialized: now 1781997710640401 us; error 48 us; skew 500 ppm
I20260620 23:21:50.640749 3531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:50 dist-test-slave-5kzz krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:50.643829 3531 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:21:50.645097 3531 webserver.cc:460] Webserver started at http://127.0.97.1:44111/ using document root <none> and password file <none>
I20260620 23:21:50.645804 3531 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:50.645900 3531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:50.646140 3531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:50.648901 3531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "038d70e165dc422b987a495aaf98e16d"
format_stamp: "Formatted at 2026-06-20 23:21:50 on dist-test-slave-5kzz"
server_key: "f739bf7089aa0ad2c7094bbeff55983b"
server_key_iv: "166ece831a321a4546ea65d58b206c42"
server_key_version: "encryptionkey@0"
I20260620 23:21:50.649611 3531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "038d70e165dc422b987a495aaf98e16d"
format_stamp: "Formatted at 2026-06-20 23:21:50 on dist-test-slave-5kzz"
server_key: "f739bf7089aa0ad2c7094bbeff55983b"
server_key_iv: "166ece831a321a4546ea65d58b206c42"
server_key_version: "encryptionkey@0"
I20260620 23:21:50.654733 3531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:21:50.658311 3546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:50.659644 3531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:21:50.659768 3531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "038d70e165dc422b987a495aaf98e16d"
format_stamp: "Formatted at 2026-06-20 23:21:50 on dist-test-slave-5kzz"
server_key: "f739bf7089aa0ad2c7094bbeff55983b"
server_key_iv: "166ece831a321a4546ea65d58b206c42"
server_key_version: "encryptionkey@0"
I20260620 23:21:50.659916 3531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:50.684715 3531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:50.689149 3531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:50.689376 3531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:50.690227 3531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:50.691540 3531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:50.691605 3531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:50.691682 3531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:50.691723 3531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:50.724090 3531 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38731
I20260620 23:21:50.724175 3658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38731 every 8 connection(s)
I20260620 23:21:50.725330 3531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:21:50.727104 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 3531
I20260620 23:21:50.727319 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:21:50.728039 388 external_mini_cluster.cc:1428] Setting key dd13955aa38020f8ed236194d57fb211
Jun 20 23:21:50 dist-test-slave-5kzz krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:21:50.747859 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:50.727991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43633 (local address 127.0.97.62:33577)
0620 23:21:50.729053 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:50.729060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:50.729720 (+ 660us) server_negotiation.cc:407] Connection header received
0620 23:21:50.732200 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:50.732204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:50.732275 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:50.732359 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:50.734562 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:50.735332 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:50.737194 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:50.737950 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:50.738087 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:50.741837 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:50.741842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:50.741846 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:50.741877 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:50.744531 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:50.745462 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:50.745466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:50.745469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:50.745576 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:50.746064 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:50.746068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:50.746070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:50.746340 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:50.746449 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:50.747430 (+ 981us) server_negotiation.cc:299] Negotiation successful
0620 23:21:50.747598 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":805,"threads_started":1}
I20260620 23:21:50.748941 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:50.728517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33577 (local address 127.0.97.1:43633)
0620 23:21:50.729409 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:50.729475 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:50.731865 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:50.732666 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:50.732681 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:50.733498 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:50.734350 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:50.734379 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:50.735495 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:50.735503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:50.737042 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:50.737053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:50.738151 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:50.738155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:50.738345 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:50.739098 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:50.739121 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:50.741594 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:50.744676 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:50.744689 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:50.744707 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:50.745326 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:50.745689 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:50.745697 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:50.745701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:50.745945 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:50.746480 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:50.746489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:50.746999 (+ 510us) client_negotiation.cc:769] Sending connection context
0620 23:21:50.747320 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 23:21:50.747792 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":418,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:21:50.750514 3659 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33577
I20260620 23:21:50.750873 3659 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:50.751736 3659 heartbeater.cc:511] Master 127.0.97.62:33577 requested a full tablet report, sending...
I20260620 23:21:50.753885 3470 ts_manager.cc:194] Registered new tserver with Master: 038d70e165dc422b987a495aaf98e16d (127.0.97.1:38731)
I20260620 23:21:50.755825 3470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:21:50.771119 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33577
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:51.084776 3665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:51.085067 3665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:51.085217 3665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:51.092905 3665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:51.093024 3665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:51.093159 3665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:21:51.102854 3665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:51.104144 3665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:51.105423 3665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:51.118741 3671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:51.118834 3673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:51.118805 3670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:51.119755 3665 server_base.cc:1034] running on GCE node
I20260620 23:21:51.120329 3665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:51.121208 3665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:51.122407 3665 hybrid_clock.cc:648] HybridClock initialized: now 1781997711122355 us; error 40 us; skew 500 ppm
I20260620 23:21:51.122730 3665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:51 dist-test-slave-5kzz krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:51.125882 3665 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:21:51.127420 3665 webserver.cc:460] Webserver started at http://127.0.97.2:39673/ using document root <none> and password file <none>
I20260620 23:21:51.128165 3665 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:51.128306 3665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:51.128604 3665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:51.131508 3665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "6f15f305f9e040bca1e457c784fc24d0"
format_stamp: "Formatted at 2026-06-20 23:21:51 on dist-test-slave-5kzz"
server_key: "2f67aeb38b9e2977c4702ac4a1b03398"
server_key_iv: "a6c825d576bf022f323733291f774b46"
server_key_version: "encryptionkey@0"
I20260620 23:21:51.132310 3665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6f15f305f9e040bca1e457c784fc24d0"
format_stamp: "Formatted at 2026-06-20 23:21:51 on dist-test-slave-5kzz"
server_key: "2f67aeb38b9e2977c4702ac4a1b03398"
server_key_iv: "a6c825d576bf022f323733291f774b46"
server_key_version: "encryptionkey@0"
I20260620 23:21:51.137702 3665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:21:51.141299 3680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:51.142771 3665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:21:51.142940 3665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "6f15f305f9e040bca1e457c784fc24d0"
format_stamp: "Formatted at 2026-06-20 23:21:51 on dist-test-slave-5kzz"
server_key: "2f67aeb38b9e2977c4702ac4a1b03398"
server_key_iv: "a6c825d576bf022f323733291f774b46"
server_key_version: "encryptionkey@0"
I20260620 23:21:51.143134 3665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:51.172276 3665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:51.176676 3665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:51.176980 3665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:51.178040 3665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:51.179540 3665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:51.179615 3665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:51.179756 3665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:51.179836 3665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:51.212481 3665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38805
I20260620 23:21:51.212589 3792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38805 every 8 connection(s)
I20260620 23:21:51.213809 3665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:21:51.219547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 3665
I20260620 23:21:51.219758 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:21:51.220227 388 external_mini_cluster.cc:1428] Setting key 054d8499a1b4035dee5a00ee8b9a19b2
Jun 20 23:21:51 dist-test-slave-5kzz krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:51.234527 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:51.216687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45081 (local address 127.0.97.62:33577)
0620 23:21:51.216865 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:51.216870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:51.218077 (+ 1207us) server_negotiation.cc:407] Connection header received
0620 23:21:51.220010 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:51.220016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:51.220112 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:51.220225 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:51.221806 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.222616 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.224197 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.224602 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.224741 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:51.228198 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:51.228202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:51.228206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:51.228237 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:51.230910 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:51.231753 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:51.231763 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:51.231770 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:51.231948 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:51.232405 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:51.232419 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:51.232426 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:51.232885 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:51.233055 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:51.233750 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 23:21:51.234014 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:21:51.235186 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:51.217221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33577 (local address 127.0.97.2:45081)
0620 23:21:51.217917 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:51.217954 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:51.219675 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:51.220463 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:51.220472 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:51.220983 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:51.221640 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:51.221658 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.222808 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.222814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:51.224043 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:51.224055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.224830 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.224837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:51.225082 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:51.225766 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:51.225790 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:51.227955 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:51.231064 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:51.231074 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:51.231092 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:51.231609 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:51.232086 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:51.232090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:51.232092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:51.232270 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:51.233017 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:51.233023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:51.233374 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 23:21:51.233614 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 23:21:51.234011 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":294,"threads_started":1}
I20260620 23:21:51.236707 3793 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33577
I20260620 23:21:51.237078 3793 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:51.238095 3793 heartbeater.cc:511] Master 127.0.97.62:33577 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:21:51.239773 3470 ts_manager.cc:194] Registered new tserver with Master: 6f15f305f9e040bca1e457c784fc24d0 (127.0.97.2:38805)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 23:21:51.240954 3470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:21:51.262084 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33577
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:51.578459 3798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:51.578871 3798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:51.579056 3798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:51.587301 3798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:51.587446 3798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:51.587630 3798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:21:51.597324 3798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:51.598654 3798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:51.599990 3798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:51.612664 3803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:51.612592 3804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:51.612605 3806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:51.613482 3798 server_base.cc:1034] running on GCE node
I20260620 23:21:51.614130 3798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:51.615082 3798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:51.616304 3798 hybrid_clock.cc:648] HybridClock initialized: now 1781997711616255 us; error 44 us; skew 500 ppm
I20260620 23:21:51.616619 3798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:51 dist-test-slave-5kzz krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:51.619599 3798 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:21:51.620975 3798 webserver.cc:460] Webserver started at http://127.0.97.3:40125/ using document root <none> and password file <none>
I20260620 23:21:51.621637 3798 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:51.621770 3798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:51.622064 3798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:51.624881 3798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "a48b9b0d194b430586fac1ae20220f57"
format_stamp: "Formatted at 2026-06-20 23:21:51 on dist-test-slave-5kzz"
server_key: "963c2e3398ab667815a95cea7aff9930"
server_key_iv: "35e35f40877c4462d150f8a4ede649cd"
server_key_version: "encryptionkey@0"
I20260620 23:21:51.625627 3798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a48b9b0d194b430586fac1ae20220f57"
format_stamp: "Formatted at 2026-06-20 23:21:51 on dist-test-slave-5kzz"
server_key: "963c2e3398ab667815a95cea7aff9930"
server_key_iv: "35e35f40877c4462d150f8a4ede649cd"
server_key_version: "encryptionkey@0"
I20260620 23:21:51.630429 3798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 23:21:51.633644 3813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:51.635032 3798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:21:51.635197 3798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "a48b9b0d194b430586fac1ae20220f57"
format_stamp: "Formatted at 2026-06-20 23:21:51 on dist-test-slave-5kzz"
server_key: "963c2e3398ab667815a95cea7aff9930"
server_key_iv: "35e35f40877c4462d150f8a4ede649cd"
server_key_version: "encryptionkey@0"
I20260620 23:21:51.635397 3798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:51.647538 3798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:51.651260 3798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:51.651556 3798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:51.652426 3798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:51.653683 3798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:51.653780 3798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:51.653951 3798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:51.654057 3798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:51.685613 3798 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40269
I20260620 23:21:51.685691 3925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40269 every 8 connection(s)
I20260620 23:21:51.686915 3798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:21:51.690434 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 3798
I20260620 23:21:51.690606 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:21:51.691218 388 external_mini_cluster.cc:1428] Setting key bc160419b2814c523f8376c050d5b31a
Jun 20 23:21:51 dist-test-slave-5kzz krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:51.706566 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:51.689235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54005 (local address 127.0.97.62:33577)
0620 23:21:51.689440 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:51.689447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:51.690752 (+ 1305us) server_negotiation.cc:407] Connection header received
0620 23:21:51.693221 (+ 2469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:51.693226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:51.693316 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:51.693414 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:51.695052 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.695765 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.697159 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.697521 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.697601 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:51.700854 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:51.700858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:51.700862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:51.700891 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:51.703334 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:51.703987 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:51.703990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:51.703993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:51.704127 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:51.704650 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:51.704655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:51.704658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:51.705011 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:51.705170 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:51.706019 (+ 849us) server_negotiation.cc:299] Negotiation successful
0620 23:21:51.706234 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:21:51.707492 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:51.689752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33577 (local address 127.0.97.3:54005)
0620 23:21:51.690549 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:51.690609 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:51.692877 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:51.693613 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:51.693620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:51.694292 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:51.694914 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:51.694932 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.695883 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.695886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:51.697036 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:51.697046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.697688 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.697692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:51.697862 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:51.698509 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:51.698530 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:51.700670 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:51.703480 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:51.703486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:51.703499 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:51.703883 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:51.704214 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:51.704217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:51.704219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:51.704453 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:51.705159 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:51.705169 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:51.705590 (+ 421us) client_negotiation.cc:769] Sending connection context
0620 23:21:51.705923 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 23:21:51.706415 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":290,"threads_started":1}
I20260620 23:21:51.708663 3926 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33577
I20260620 23:21:51.708978 3926 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:51.709708 3926 heartbeater.cc:511] Master 127.0.97.62:33577 requested a full tablet report, sending...
I20260620 23:21:51.711090 3470 ts_manager.cc:194] Registered new tserver with Master: a48b9b0d194b430586fac1ae20220f57 (127.0.97.3:40269)
I20260620 23:21:51.711848 3470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54005
I20260620 23:21:51.718582 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:21:51 dist-test-slave-5kzz krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997711, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260620 23:21:51.758483 3659 heartbeater.cc:503] Master 127.0.97.62:33577 was elected leader, sending a full tablet report...
Jun 20 23:21:51 dist-test-slave-5kzz krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997711, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:51.765278 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:51.751397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.0.97.62:33577)
0620 23:21:51.751568 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:51.751572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:51.752000 (+ 428us) server_negotiation.cc:407] Connection header received
0620 23:21:51.752155 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:51.752159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:51.752219 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:51.752310 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:51.753583 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.754397 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.755634 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.755990 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.756082 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:51.760024 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:51.760033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:51.760042 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:51.760072 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:51.762637 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:51.763339 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:51.763346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:51.763349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:51.763509 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:51.763901 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:51.763906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:51.763909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:51.764232 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:51.764464 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:51.764738 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 23:21:51.764951 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 23:21:51.765974 3470 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45958
I20260620 23:21:51.767691 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3531
Jun 20 23:21:51 dist-test-slave-5kzz krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997711, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:21:51.782287 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:51.768637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.0.97.1:38731)
0620 23:21:51.769221 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:51.769228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:51.769251 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:21:51.769306 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:51.769312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:51.769503 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:51.769642 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:51.770624 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.771613 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.772763 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.773329 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.773423 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:51.777081 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:51.777091 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:51.777108 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:51.777169 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:51.780101 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:51.780692 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:51.780699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:51.780701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:51.780810 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:51.781122 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:51.781125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:51.781127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:51.781421 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:51.781628 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:51.781833 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 23:21:51.782014 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":299,"threads_started":1}
W20260620 23:21:51.783519 3654 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41718
W20260620 23:21:51.784654 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3531: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:21:51.784857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 3531
I20260620 23:21:51.800211 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3665
Jun 20 23:21:51 dist-test-slave-5kzz krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997711, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:21:51.815613 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:51.801255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.0.97.2:38805)
0620 23:21:51.801752 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:51.801760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:51.801783 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:21:51.801837 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:51.801843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:51.802054 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:51.802178 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:51.803354 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.804352 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.805494 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.806048 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.806154 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:51.810130 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:51.810140 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:51.810154 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:51.810185 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:51.812876 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:51.813422 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:51.813428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:51.813430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:51.813550 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:51.813877 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:51.813882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:51.813884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:51.814258 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:51.814561 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:51.814942 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 23:21:51.815171 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":253,"threads_started":1}
W20260620 23:21:51.816839 3788 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44556
W20260620 23:21:51.818022 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3665: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:21:51.818151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 3665
I20260620 23:21:51.833245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3798
Jun 20 23:21:51 dist-test-slave-5kzz krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997711, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:21:51.848420 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:51.834158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46650 (local address 127.0.97.3:40269)
0620 23:21:51.834872 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:51.834879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:51.834908 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:21:51.834964 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:51.834970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:51.835185 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:51.835313 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:51.836498 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.837523 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.838669 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:51.839284 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:51.839398 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:51.843440 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:51.843450 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:51.843463 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:51.843492 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:51.845980 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:51.846558 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:51.846564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:51.846567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:51.846670 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:51.847045 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:51.847048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:51.847050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:51.847343 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:51.847559 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:51.847846 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 23:21:51.848039 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":348,"threads_started":1}
W20260620 23:21:51.849592 3921 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46650
W20260620 23:21:51.850855 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3798: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:21:51.851030 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 3798
I20260620 23:21:51.866393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3438
I20260620 23:21:51.867614 3500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:51.968060 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 3438
2026-06-20T23:21:51Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3535 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:21:52 dist-test-slave-5kzz krb5kdc[3943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:21:52 dist-test-slave-5kzz krb5kdc[3943](info): set up 2 sockets
Jun 20 23:21:52 dist-test-slave-5kzz krb5kdc[3943](info): commencing operation
krb5kdc: starting...
W20260620 23:21:53.059042 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.010s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:21:53 dist-test-slave-5kzz krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:21:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:21:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:21:53.239111 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46467
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:53.545392 3956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:53.545725 3956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:53.545879 3956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:53.552942 3956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:21:53.553108 3956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:53.553236 3956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:53.553346 3956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:21:53.553421 3956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:21:53.563289 3956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:53.564616 3956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:53.565886 3956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:53.582755 3962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:53.582885 3961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:53.583786 3956 server_base.cc:1034] running on GCE node
W20260620 23:21:53.583827 3964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:53.584681 3956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:53.586047 3956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:53.587245 3956 hybrid_clock.cc:648] HybridClock initialized: now 1781997713587180 us; error 58 us; skew 500 ppm
I20260620 23:21:53.587615 3956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:53 dist-test-slave-5kzz krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:53.593511 3956 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:21:53.595127 3956 webserver.cc:460] Webserver started at http://127.0.97.62:35491/ using document root <none> and password file <none>
I20260620 23:21:53.595777 3956 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:53.595873 3956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:53.596128 3956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:53.599104 3956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "54320dc8a29b48d3beb6403da1d6fcd0"
format_stamp: "Formatted at 2026-06-20 23:21:53 on dist-test-slave-5kzz"
server_key: "c0d88f4fafe6c2e76e133f262f1b5269"
server_key_iv: "dfa601779ed228c45a4ae58c1d73927f"
server_key_version: "encryptionkey@0"
I20260620 23:21:53.599807 3956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "54320dc8a29b48d3beb6403da1d6fcd0"
format_stamp: "Formatted at 2026-06-20 23:21:53 on dist-test-slave-5kzz"
server_key: "c0d88f4fafe6c2e76e133f262f1b5269"
server_key_iv: "dfa601779ed228c45a4ae58c1d73927f"
server_key_version: "encryptionkey@0"
I20260620 23:21:53.604790 3956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:21:53.608114 3971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:53.609383 3956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:21:53.609575 3956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "54320dc8a29b48d3beb6403da1d6fcd0"
format_stamp: "Formatted at 2026-06-20 23:21:53 on dist-test-slave-5kzz"
server_key: "c0d88f4fafe6c2e76e133f262f1b5269"
server_key_iv: "dfa601779ed228c45a4ae58c1d73927f"
server_key_version: "encryptionkey@0"
I20260620 23:21:53.609760 3956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:53.637923 3956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:53.641762 3956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:53.642052 3956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:53.659749 3956 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46467
I20260620 23:21:53.659771 4022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46467 every 8 connection(s)
I20260620 23:21:53.661051 3956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:21:53.664544 4023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:53.666322 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 3956
I20260620 23:21:53.666530 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:21:53.667301 388 external_mini_cluster.cc:1428] Setting key eaf2a56585cce8cd4439150c05317843
I20260620 23:21:53.677287 4023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0: Bootstrap starting.
I20260620 23:21:53.680209 4023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0: Neither blocks nor log segments found. Creating new log.
Jun 20 23:21:53 dist-test-slave-5kzz krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:53.681236 4023 log.cc:824] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:53.684072 4023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0: No bootstrap required, opened a new log
I20260620 23:21:53.688392 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:53.669360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49696 (local address 127.0.97.62:46467)
0620 23:21:53.670102 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:53.670113 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:53.670155 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:21:53.671775 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:53.671799 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:53.672190 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:53.672552 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:53.673984 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:53.675368 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:53.676731 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:53.677406 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:53.677606 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:53.681339 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:53.681354 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:53.681379 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:53.681426 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:53.684502 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:53.685144 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:53.685156 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:53.685161 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:53.685313 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:53.685691 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:53.685700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:53.685705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:53.686219 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:53.686565 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:53.687016 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 23:21:53.687420 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":303,"threads_started":1}
I20260620 23:21:53.691931 4023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54320dc8a29b48d3beb6403da1d6fcd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46467 } }
I20260620 23:21:53.692274 4023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:53.692376 4023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54320dc8a29b48d3beb6403da1d6fcd0, State: Initialized, Role: FOLLOWER
I20260620 23:21:53.692889 4023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54320dc8a29b48d3beb6403da1d6fcd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46467 } }
I20260620 23:21:53.693094 4023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:21:53.693280 4023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:21:53.693452 4023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:53.695531 4023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54320dc8a29b48d3beb6403da1d6fcd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46467 } }
I20260620 23:21:53.696002 4023 leader_election.cc:302] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54320dc8a29b48d3beb6403da1d6fcd0; no voters:
I20260620 23:21:53.696534 4023 leader_election.cc:288] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:21:53.696697 4028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:53.697374 4028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [term 1 LEADER]: Becoming Leader. State: Replica: 54320dc8a29b48d3beb6403da1d6fcd0, State: Running, Role: LEADER
I20260620 23:21:53.697901 4028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54320dc8a29b48d3beb6403da1d6fcd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46467 } }
I20260620 23:21:53.698550 4023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:21:53.700827 4029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54320dc8a29b48d3beb6403da1d6fcd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54320dc8a29b48d3beb6403da1d6fcd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46467 } } }
I20260620 23:21:53.701154 4029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:53.702744 4035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:21:53.702570 4030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54320dc8a29b48d3beb6403da1d6fcd0. Latest consensus state: current_term: 1 leader_uuid: "54320dc8a29b48d3beb6403da1d6fcd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54320dc8a29b48d3beb6403da1d6fcd0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46467 } } }
I20260620 23:21:53.702857 4030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:53.707258 4035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:21:53.716965 4035 catalog_manager.cc:1347] Generated new cluster ID: 0f99f2056ab045cdb9e3a9ce7686cc0f
I20260620 23:21:53.717103 4035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:21:53.731961 4035 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:21:53.733242 4035 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:21:53.740955 4035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54320dc8a29b48d3beb6403da1d6fcd0: Generated new TSK 0
I20260620 23:21:53.742069 4035 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:21:53.799660 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46467
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:54.124965 4049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:54.125270 4049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:54.125437 4049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:54.132586 4049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:54.132750 4049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:54.132920 4049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:21:54.142602 4049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:54.143988 4049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:54.145313 4049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:54.158351 4057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:54.158500 4055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:54.158718 4054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:54.159686 4049 server_base.cc:1034] running on GCE node
I20260620 23:21:54.160326 4049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:54.161243 4049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:54.162465 4049 hybrid_clock.cc:648] HybridClock initialized: now 1781997714162405 us; error 58 us; skew 500 ppm
I20260620 23:21:54.162827 4049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:54 dist-test-slave-5kzz krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:54.165774 4049 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:21:54.167248 4049 webserver.cc:460] Webserver started at http://127.0.97.1:44567/ using document root <none> and password file <none>
I20260620 23:21:54.167920 4049 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:54.168057 4049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:54.168375 4049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:54.171254 4049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "ba5b0976892f4d8cb30c9a9be6e3fc77"
format_stamp: "Formatted at 2026-06-20 23:21:54 on dist-test-slave-5kzz"
server_key: "9643496724a778f4ee3d381d45e34228"
server_key_iv: "dca5579de20c2d61945e641b74e90c3f"
server_key_version: "encryptionkey@0"
I20260620 23:21:54.172055 4049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ba5b0976892f4d8cb30c9a9be6e3fc77"
format_stamp: "Formatted at 2026-06-20 23:21:54 on dist-test-slave-5kzz"
server_key: "9643496724a778f4ee3d381d45e34228"
server_key_iv: "dca5579de20c2d61945e641b74e90c3f"
server_key_version: "encryptionkey@0"
I20260620 23:21:54.177090 4049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 23:21:54.180303 4064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:54.181452 4049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:21:54.181615 4049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "ba5b0976892f4d8cb30c9a9be6e3fc77"
format_stamp: "Formatted at 2026-06-20 23:21:54 on dist-test-slave-5kzz"
server_key: "9643496724a778f4ee3d381d45e34228"
server_key_iv: "dca5579de20c2d61945e641b74e90c3f"
server_key_version: "encryptionkey@0"
I20260620 23:21:54.181813 4049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:54.199788 4049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:54.203526 4049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:54.203811 4049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:54.204690 4049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:54.205951 4049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:54.206060 4049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:54.206199 4049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:54.206271 4049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:54.238061 4049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44939
I20260620 23:21:54.238104 4176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44939 every 8 connection(s)
I20260620 23:21:54.239368 4049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:21:54.240324 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 4049
I20260620 23:21:54.240576 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:21:54.241389 388 external_mini_cluster.cc:1428] Setting key bc69634d0e8d52dec41712376fc96802
Jun 20 23:21:54 dist-test-slave-5kzz krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:21:54.259187 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:54.242080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49303 (local address 127.0.97.62:46467)
0620 23:21:54.242575 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:54.242579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:54.243680 (+ 1101us) server_negotiation.cc:407] Connection header received
0620 23:21:54.245388 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:54.245393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:54.245471 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:54.245580 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:54.247073 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:54.247751 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:54.249183 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:54.249559 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:54.249689 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:54.253359 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:54.253363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:54.253367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:54.253395 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:54.255898 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:54.256834 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:54.256838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:54.256840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:54.256942 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:54.257402 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:54.257404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:54.257406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:54.257655 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:54.257759 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:54.258662 (+ 903us) server_negotiation.cc:299] Negotiation successful
0620 23:21:54.258900 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":284,"threads_started":1}
I20260620 23:21:54.260428 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:54.242698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46467 (local address 127.0.97.1:49303)
0620 23:21:54.243533 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:54.243582 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:54.245133 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:54.245811 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:54.245819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:54.246308 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:54.246923 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:54.246941 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:54.247888 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:54.247895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:54.249037 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:54.249048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:54.249771 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:54.249778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:54.249994 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:54.250623 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:54.250650 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:54.253122 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:54.256049 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:54.256060 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:54.256080 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:54.256713 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:54.257046 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:54.257052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:54.257056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:54.257291 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:54.257780 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:54.257789 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:54.258261 (+ 472us) client_negotiation.cc:769] Sending connection context
0620 23:21:54.258569 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 23:21:54.259175 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":296,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:21:54.262034 4177 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46467
I20260620 23:21:54.262346 4177 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:54.263127 4177 heartbeater.cc:511] Master 127.0.97.62:46467 requested a full tablet report, sending...
I20260620 23:21:54.265334 3988 ts_manager.cc:194] Registered new tserver with Master: ba5b0976892f4d8cb30c9a9be6e3fc77 (127.0.97.1:44939)
I20260620 23:21:54.266868 3988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:21:54.283450 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46467
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:54.600266 4183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:54.600571 4183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:54.600742 4183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:54.607717 4183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:54.607865 4183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:54.608028 4183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:21:54.617870 4183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:54.619088 4183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:54.620394 4183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:54.632845 4188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:54.632956 4191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:54.632936 4189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:54.633795 4183 server_base.cc:1034] running on GCE node
I20260620 23:21:54.634344 4183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:54.635263 4183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:54.636447 4183 hybrid_clock.cc:648] HybridClock initialized: now 1781997714636387 us; error 50 us; skew 500 ppm
I20260620 23:21:54.636734 4183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:54 dist-test-slave-5kzz krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:54.639716 4183 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:21:54.641124 4183 webserver.cc:460] Webserver started at http://127.0.97.2:43613/ using document root <none> and password file <none>
I20260620 23:21:54.641755 4183 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:54.641889 4183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:54.642204 4183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:54.645015 4183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "b9c7b6fc73fc442f9a10b9364b192b84"
format_stamp: "Formatted at 2026-06-20 23:21:54 on dist-test-slave-5kzz"
server_key: "3102a1a2c16737092bf0e96f5687efa1"
server_key_iv: "b567f61891a52698ed2ac4ce9ad3df53"
server_key_version: "encryptionkey@0"
I20260620 23:21:54.645757 4183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b9c7b6fc73fc442f9a10b9364b192b84"
format_stamp: "Formatted at 2026-06-20 23:21:54 on dist-test-slave-5kzz"
server_key: "3102a1a2c16737092bf0e96f5687efa1"
server_key_iv: "b567f61891a52698ed2ac4ce9ad3df53"
server_key_version: "encryptionkey@0"
I20260620 23:21:54.650571 4183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 23:21:54.653784 4198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:54.655069 4183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:21:54.655241 4183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "b9c7b6fc73fc442f9a10b9364b192b84"
format_stamp: "Formatted at 2026-06-20 23:21:54 on dist-test-slave-5kzz"
server_key: "3102a1a2c16737092bf0e96f5687efa1"
server_key_iv: "b567f61891a52698ed2ac4ce9ad3df53"
server_key_version: "encryptionkey@0"
I20260620 23:21:54.655438 4183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:54.674577 4183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:54.678184 4183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:54.678464 4183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:54.679404 4183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:54.680724 4183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:54.680824 4183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:54.680970 4183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:54.681043 4183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:54.714488 4183 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37071
I20260620 23:21:54.714531 4310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37071 every 8 connection(s)
I20260620 23:21:54.715764 4183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:21:54.721258 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 4183
I20260620 23:21:54.721449 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:21:54.721899 388 external_mini_cluster.cc:1428] Setting key 1b288b88eb4d1d2301dac3457cadc58b
Jun 20 23:21:54 dist-test-slave-5kzz krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:54.736989 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:54.718102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59119 (local address 127.0.97.62:46467)
0620 23:21:54.718302 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:54.718308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:54.719546 (+ 1238us) server_negotiation.cc:407] Connection header received
0620 23:21:54.721211 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:54.721216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:54.721299 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:54.721415 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:54.723157 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:54.724084 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:54.726592 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:54.727042 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:54.727143 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:54.731565 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:54.731570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:54.731574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:54.731602 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:54.734081 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:54.734801 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:54.734813 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:54.734816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:54.734939 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:54.735359 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:54.735367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:54.735372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:54.735628 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:54.735787 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:54.736436 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 23:21:54.736649 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:21:54.737914 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:54.718726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46467 (local address 127.0.97.2:59119)
0620 23:21:54.719395 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:54.719431 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:54.720925 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:54.721782 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:54.721790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:54.722322 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:54.722992 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:54.723010 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:54.724240 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:54.724247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:54.726451 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:54.726468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:54.727258 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:54.727268 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:54.727607 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:54.728330 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:54.728370 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:54.731359 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:54.734222 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:54.734231 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:54.734247 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:54.734653 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:54.735047 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:54.735052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:54.735056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:54.735230 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:54.735754 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:54.735761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:54.736085 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 23:21:54.736346 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 23:21:54.736846 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":269,"threads_started":1}
I20260620 23:21:54.739418 4311 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46467
I20260620 23:21:54.739701 4311 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:54.740387 4311 heartbeater.cc:511] Master 127.0.97.62:46467 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:21:54.741870 3988 ts_manager.cc:194] Registered new tserver with Master: b9c7b6fc73fc442f9a10b9364b192b84 (127.0.97.2:37071)
I20260620 23:21:54.742609 3988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:21:54.762804 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46467
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:55.071143 4316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:55.071421 4316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:55.071646 4316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:55.078828 4316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:55.078931 4316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:55.079138 4316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:21:55.089416 4316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:55.090732 4316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:55.092010 4316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:55.105031 4321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:55.105082 4324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:55.105129 4322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:55.105952 4316 server_base.cc:1034] running on GCE node
I20260620 23:21:55.106485 4316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:55.107398 4316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:55.108582 4316 hybrid_clock.cc:648] HybridClock initialized: now 1781997715108532 us; error 46 us; skew 500 ppm
I20260620 23:21:55.108834 4316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:55 dist-test-slave-5kzz krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:55.111714 4316 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:21:55.113152 4316 webserver.cc:460] Webserver started at http://127.0.97.3:43105/ using document root <none> and password file <none>
I20260620 23:21:55.113983 4316 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:55.114128 4316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:55.114431 4316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:55.117309 4316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "2d3477345614413ba7302be4843afeb2"
format_stamp: "Formatted at 2026-06-20 23:21:55 on dist-test-slave-5kzz"
server_key: "6211f1e4e94634d60cb2aecae3ac44d3"
server_key_iv: "ac857f1490bdef67846efd978c6172fb"
server_key_version: "encryptionkey@0"
I20260620 23:21:55.118060 4316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2d3477345614413ba7302be4843afeb2"
format_stamp: "Formatted at 2026-06-20 23:21:55 on dist-test-slave-5kzz"
server_key: "6211f1e4e94634d60cb2aecae3ac44d3"
server_key_iv: "ac857f1490bdef67846efd978c6172fb"
server_key_version: "encryptionkey@0"
I20260620 23:21:55.122965 4316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:21:55.126072 4331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:55.127390 4316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:21:55.127559 4316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "2d3477345614413ba7302be4843afeb2"
format_stamp: "Formatted at 2026-06-20 23:21:55 on dist-test-slave-5kzz"
server_key: "6211f1e4e94634d60cb2aecae3ac44d3"
server_key_iv: "ac857f1490bdef67846efd978c6172fb"
server_key_version: "encryptionkey@0"
I20260620 23:21:55.127764 4316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:55.156687 4316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:55.160642 4316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:55.160910 4316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:55.161878 4316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:55.163203 4316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:55.163311 4316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:55.163446 4316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:55.163527 4316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:55.194470 4316 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34491
I20260620 23:21:55.194517 4443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34491 every 8 connection(s)
I20260620 23:21:55.195719 4316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:21:55.199599 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 4316
I20260620 23:21:55.199812 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:21:55.200338 388 external_mini_cluster.cc:1428] Setting key 483bdbcec36c1efc269884e0c9866ef9
Jun 20 23:21:55 dist-test-slave-5kzz krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:55.215853 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:55.198402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57929 (local address 127.0.97.62:46467)
0620 23:21:55.198611 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:55.198631 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:55.200280 (+ 1649us) server_negotiation.cc:407] Connection header received
0620 23:21:55.202589 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:55.202594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:55.202672 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:55.202781 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:55.204323 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.205073 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.206556 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.207033 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.207175 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:55.210235 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:55.210241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:55.210245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:55.210283 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:55.212726 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:55.213536 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:55.213541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:55.213544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:55.213678 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:55.214135 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:55.214141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:55.214144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:55.214417 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:55.214568 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:55.215295 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 23:21:55.215507 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:21:55.216717 4446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:55.199162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46467 (local address 127.0.97.3:57929)
0620 23:21:55.200121 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:55.200166 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:55.202355 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:55.203035 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:55.203045 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:55.203651 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:55.204172 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:55.204187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.205218 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.205224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:55.206419 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:55.206429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.207255 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.207263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:55.207485 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:55.208115 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:55.208138 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:55.210057 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:55.212856 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:55.212864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:55.212878 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:55.213344 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:55.213792 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:55.213797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:55.213800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:55.214036 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:55.214547 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:55.214555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:55.214945 (+ 390us) client_negotiation.cc:769] Sending connection context
0620 23:21:55.215190 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 23:21:55.215686 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":403,"threads_started":1}
I20260620 23:21:55.217911 4444 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46467
I20260620 23:21:55.218199 4444 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:55.218922 4444 heartbeater.cc:511] Master 127.0.97.62:46467 requested a full tablet report, sending...
I20260620 23:21:55.220153 3988 ts_manager.cc:194] Registered new tserver with Master: 2d3477345614413ba7302be4843afeb2 (127.0.97.3:34491)
I20260620 23:21:55.220865 3988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57929
I20260620 23:21:55.227300 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:21:55 dist-test-slave-5kzz krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:21:55 dist-test-slave-5kzz krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:21:55.270399 4177 heartbeater.cc:503] Master 127.0.97.62:46467 was elected leader, sending a full tablet report...
I20260620 23:21:55.272286 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:55.258372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.0.97.1:44939)
0620 23:21:55.258962 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:55.258970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:55.259003 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:21:55.259065 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:55.259072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:55.259271 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:55.259391 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:55.260695 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.261627 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.262896 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.263400 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.263492 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:55.266928 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:55.266937 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:55.266949 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:55.266978 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:55.269448 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:55.270000 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:55.270006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:55.270009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:55.270125 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:55.270633 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:55.270636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:55.270638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:55.271086 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:55.271395 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:55.271702 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 23:21:55.271949 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":327,"threads_started":1}
Jun 20 23:21:55 dist-test-slave-5kzz krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:55.287640 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:55.276033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.0.97.62:46467)
0620 23:21:55.276253 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:55.276259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:55.276375 (+ 116us) server_negotiation.cc:407] Connection header received
0620 23:21:55.276478 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:55.276481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:55.276535 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:55.276608 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:55.277602 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.278271 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.279431 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.279795 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.279888 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:55.283294 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:55.283298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:55.283301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:55.283326 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:55.285554 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:55.286084 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:55.286091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:55.286107 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:55.286223 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:55.286490 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:55.286493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:55.286494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:55.286821 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:55.286990 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:55.287216 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 23:21:55.287350 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260620 23:21:55.288160 3988 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49706
I20260620 23:21:55.288990 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 4049
I20260620 23:21:55.300710 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:55.289848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57430 (local address 127.0.97.1:44939)
0620 23:21:55.290030 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:55.290035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:55.290220 (+ 185us) server_negotiation.cc:407] Connection header received
0620 23:21:55.290382 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:55.290387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:55.290449 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:55.290542 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:55.291648 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.292271 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.293460 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.293860 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.293984 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:55.296016 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:55.296021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:55.296024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:55.296063 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:55.298258 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:55.298885 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:55.298891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:55.298894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:55.299049 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:55.299408 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:55.299412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:55.299416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:55.299636 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:55.299870 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:55.300206 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 23:21:55.300411 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:21:55.301257 4172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:55.380180 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 4049
I20260620 23:21:55.395639 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 4183
Jun 20 23:21:55 dist-test-slave-5kzz krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:21:55.410655 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:55.396508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.0.97.2:37071)
0620 23:21:55.397066 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:55.397073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:55.397098 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:21:55.397156 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:55.397162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:55.397409 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:55.397535 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:55.398617 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.399758 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.400864 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.401362 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.401453 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:55.405169 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:55.405179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:55.405193 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:55.405228 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:55.408003 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:55.408589 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:55.408603 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:55.408606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:55.408801 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:55.409123 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:55.409127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:55.409130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:55.409648 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:55.409969 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:55.410085 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 23:21:55.410326 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":303,"threads_started":1}
I20260620 23:21:55.411963 4306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:55.492635 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 4183
I20260620 23:21:55.508317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 4316
Jun 20 23:21:55 dist-test-slave-5kzz krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:21:55.524307 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:55.509409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.0.97.3:34491)
0620 23:21:55.510017 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:55.510024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:55.510051 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:21:55.510108 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:55.510114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:55.510316 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:55.510445 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:55.511675 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.512694 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.514212 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:55.514860 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:55.515015 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:55.518982 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:55.518992 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:55.519005 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:55.519033 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:55.521546 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:55.522161 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:55.522171 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:55.522174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:55.522292 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:55.522634 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:55.522651 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:55.522654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:55.523134 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:55.523440 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:55.523728 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 23:21:55.523978 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":342,"threads_started":1}
I20260620 23:21:55.525566 4439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:55.608745 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 4316
I20260620 23:21:55.624538 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 3956
I20260620 23:21:55.625687 4018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:55.723469 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 3956
2026-06-20T23:21:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3752 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:21:55 dist-test-slave-5kzz krb5kdc[4463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:21:55 dist-test-slave-5kzz krb5kdc[4463](info): set up 2 sockets
Jun 20 23:21:55 dist-test-slave-5kzz krb5kdc[4463](info): commencing operation
krb5kdc: starting...
W20260620 23:21:56.783030 388 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:21:56 dist-test-slave-5kzz krb5kdc[4463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:21:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:21:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:21:56.961547 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41595
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:57.268074 4476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:57.268375 4476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:57.268534 4476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:57.275394 4476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:21:57.275514 4476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:57.275626 4476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:57.275729 4476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:21:57.275820 4476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:21:57.285506 4476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:57.286756 4476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:57.288033 4476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:57.298847 4481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:57.298822 4482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:57.298779 4484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:57.299731 4476 server_base.cc:1034] running on GCE node
I20260620 23:21:57.300391 4476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:57.301612 4476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:57.302776 4476 hybrid_clock.cc:648] HybridClock initialized: now 1781997717302746 us; error 26 us; skew 500 ppm
I20260620 23:21:57.303046 4476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:57 dist-test-slave-5kzz krb5kdc[4463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:57.305902 4476 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:21:57.306983 4476 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:57.307065 4476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:57.307319 4476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:57.310050 4476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "36df571dc2624300906cad0104d09ad3"
format_stamp: "Formatted at 2026-06-20 23:21:57 on dist-test-slave-5kzz"
server_key: "168aefc22c894be510dd6cd657b9b4ad"
server_key_iv: "ca71226aad4eaf35d8cb3ad6d856651f"
server_key_version: "encryptionkey@0"
I20260620 23:21:57.310787 4476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "36df571dc2624300906cad0104d09ad3"
format_stamp: "Formatted at 2026-06-20 23:21:57 on dist-test-slave-5kzz"
server_key: "168aefc22c894be510dd6cd657b9b4ad"
server_key_iv: "ca71226aad4eaf35d8cb3ad6d856651f"
server_key_version: "encryptionkey@0"
I20260620 23:21:57.315375 4476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:21:57.318504 4490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:57.319856 4476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:21:57.319993 4476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "36df571dc2624300906cad0104d09ad3"
format_stamp: "Formatted at 2026-06-20 23:21:57 on dist-test-slave-5kzz"
server_key: "168aefc22c894be510dd6cd657b9b4ad"
server_key_iv: "ca71226aad4eaf35d8cb3ad6d856651f"
server_key_version: "encryptionkey@0"
I20260620 23:21:57.320165 4476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:57.351939 4476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:57.355738 4476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:57.355942 4476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:57.373261 4476 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41595
I20260620 23:21:57.373281 4541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41595 every 8 connection(s)
I20260620 23:21:57.374073 4476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:21:57.376905 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 4476
I20260620 23:21:57.377130 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:21:57.377668 4542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:57.377769 388 external_mini_cluster.cc:1428] Setting key 3ca0c5e806a361cf3af746fc7d939e87
I20260620 23:21:57.391440 4542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3: Bootstrap starting.
Jun 20 23:21:57 dist-test-slave-5kzz krb5kdc[4463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:57.394341 4542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:57.395423 4542 log.cc:824] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:57.397867 4542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3: No bootstrap required, opened a new log
I20260620 23:21:57.399394 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:57.379677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.0.97.62:41595)
0620 23:21:57.380463 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:57.380477 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:57.380515 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:21:57.382097 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:57.382116 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:57.382460 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:57.382864 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:57.384551 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:57.385804 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:57.387587 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:57.388253 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:57.388375 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:57.392772 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:57.392779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:57.392794 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:57.392824 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:57.395775 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:57.396423 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:57.396428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:57.396434 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:57.396611 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:57.396958 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:57.396966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:57.396968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:57.397424 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:57.397696 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:57.398110 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 23:21:57.398475 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":379,"threads_started":1}
I20260620 23:21:57.405524 4542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36df571dc2624300906cad0104d09ad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41595 } }
I20260620 23:21:57.405854 4542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:57.405958 4542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36df571dc2624300906cad0104d09ad3, State: Initialized, Role: FOLLOWER
I20260620 23:21:57.406523 4542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36df571dc2624300906cad0104d09ad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41595 } }
I20260620 23:21:57.406759 4542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:21:57.406886 4542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:21:57.407058 4542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:57.409291 4542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36df571dc2624300906cad0104d09ad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41595 } }
I20260620 23:21:57.409750 4542 leader_election.cc:302] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36df571dc2624300906cad0104d09ad3; no voters:
I20260620 23:21:57.410241 4542 leader_election.cc:288] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:21:57.410377 4547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:57.411096 4547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [term 1 LEADER]: Becoming Leader. State: Replica: 36df571dc2624300906cad0104d09ad3, State: Running, Role: LEADER
I20260620 23:21:57.411592 4547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36df571dc2624300906cad0104d09ad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41595 } }
I20260620 23:21:57.412232 4542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:21:57.414548 4549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36df571dc2624300906cad0104d09ad3. Latest consensus state: current_term: 1 leader_uuid: "36df571dc2624300906cad0104d09ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36df571dc2624300906cad0104d09ad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41595 } } }
I20260620 23:21:57.415035 4549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:57.414631 4548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36df571dc2624300906cad0104d09ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36df571dc2624300906cad0104d09ad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41595 } } }
I20260620 23:21:57.415278 4548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:21:57.418133 4556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:21:57.422106 4556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:21:57.429282 4556 catalog_manager.cc:1347] Generated new cluster ID: 88b736dc760947d4b993bdd3c631afbe
I20260620 23:21:57.429376 4556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:21:57.440603 4556 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:21:57.442215 4556 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:21:57.447278 4556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3: Generated new TSK 0
I20260620 23:21:57.447995 4556 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:21:57.496631 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41595
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:57.811674 4568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:57.811981 4568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:57.812172 4568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:57.819355 4568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:57.819511 4568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:57.819700 4568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:21:57.829077 4568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:57.830299 4568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:57.831564 4568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:57.843654 4574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:57.843712 4573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:57.843750 4576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:57.844653 4568 server_base.cc:1034] running on GCE node
I20260620 23:21:57.845183 4568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:57.845994 4568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:57.847174 4568 hybrid_clock.cc:648] HybridClock initialized: now 1781997717847116 us; error 59 us; skew 500 ppm
I20260620 23:21:57.847471 4568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:57 dist-test-slave-5kzz krb5kdc[4463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:57.850487 4568 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:21:57.851550 4568 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:57.851682 4568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:57.852000 4568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:57.854758 4568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "65065b364d41402d80bcb5edd9599ee8"
format_stamp: "Formatted at 2026-06-20 23:21:57 on dist-test-slave-5kzz"
server_key: "5dbe4158f5f8ca11314073782640d296"
server_key_iv: "b40ecc905fa77dcaaacd8c76375f3d8f"
server_key_version: "encryptionkey@0"
I20260620 23:21:57.855549 4568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "65065b364d41402d80bcb5edd9599ee8"
format_stamp: "Formatted at 2026-06-20 23:21:57 on dist-test-slave-5kzz"
server_key: "5dbe4158f5f8ca11314073782640d296"
server_key_iv: "b40ecc905fa77dcaaacd8c76375f3d8f"
server_key_version: "encryptionkey@0"
I20260620 23:21:57.860280 4568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:21:57.863438 4582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:57.864717 4568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:21:57.864880 4568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "65065b364d41402d80bcb5edd9599ee8"
format_stamp: "Formatted at 2026-06-20 23:21:57 on dist-test-slave-5kzz"
server_key: "5dbe4158f5f8ca11314073782640d296"
server_key_iv: "b40ecc905fa77dcaaacd8c76375f3d8f"
server_key_version: "encryptionkey@0"
I20260620 23:21:57.865087 4568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:57.900277 4568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:57.904558 4568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:57.904825 4568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:57.905792 4568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:57.907063 4568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:57.907193 4568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:57.907306 4568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:57.907384 4568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:57.939901 4568 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35245
I20260620 23:21:57.939945 4694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35245 every 8 connection(s)
I20260620 23:21:57.940722 4568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:21:57.945864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 4568
I20260620 23:21:57.946063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:21:57.946866 388 external_mini_cluster.cc:1428] Setting key 77946b72dfd2e03b1b6a59520c6af8bc
Jun 20 23:21:57 dist-test-slave-5kzz krb5kdc[4463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:57.962903 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:57.943055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59063 (local address 127.0.97.62:41595)
0620 23:21:57.943516 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:57.943521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:57.944436 (+ 915us) server_negotiation.cc:407] Connection header received
0620 23:21:57.946193 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:57.946199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:57.946286 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:57.946439 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:57.948175 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:57.948995 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:57.951401 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:57.951827 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:57.951919 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:57.956431 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:57.956439 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:57.956448 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:57.956491 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:57.959820 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:57.960467 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:57.960476 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:57.960482 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:57.960618 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:57.961031 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:57.961039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:57.961045 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:57.961391 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:57.961532 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:57.962166 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 23:21:57.962404 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":249,"threads_started":1}
I20260620 23:21:57.963562 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:57.943647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41595 (local address 127.0.97.1:59063)
0620 23:21:57.944271 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:57.944306 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:57.945884 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:57.946666 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:57.946713 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:57.947284 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:57.947996 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:57.948017 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:57.949182 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:57.949186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:57.951269 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:57.951288 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:57.952042 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:57.952050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:57.952370 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:57.953081 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:57.953113 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:57.956221 (+ 3108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:57.959945 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:57.959951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:57.959963 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:57.960336 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:57.960742 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:57.960746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:57.960748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:57.960906 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:57.961507 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:57.961512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:57.961793 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 23:21:57.962002 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 23:21:57.962337 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":246,"threads_started":1}
I20260620 23:21:57.965214 4695 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41595
I20260620 23:21:57.965502 4695 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:57.966099 4695 heartbeater.cc:511] Master 127.0.97.62:41595 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:21:57.968165 4507 ts_manager.cc:194] Registered new tserver with Master: 65065b364d41402d80bcb5edd9599ee8 (127.0.97.1:35245)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:21:57.969851 4507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:21:57.989472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41595
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:58.305006 4701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:58.305275 4701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:58.305460 4701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:58.312376 4701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:58.312525 4701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:58.312667 4701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:21:58.322371 4701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:58.323670 4701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:58.325019 4701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:58.337517 4709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:58.337553 4707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:58.337517 4706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:58.343811 4701 server_base.cc:1034] running on GCE node
I20260620 23:21:58.344380 4701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:58.345296 4701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:58.346488 4701 hybrid_clock.cc:648] HybridClock initialized: now 1781997718346444 us; error 42 us; skew 500 ppm
I20260620 23:21:58.346818 4701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:58 dist-test-slave-5kzz krb5kdc[4463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:58.349728 4701 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:21:58.350993 4701 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:58.351105 4701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:58.351434 4701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:58.354254 4701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "1166c2c818374309a8c0287077db16ab"
format_stamp: "Formatted at 2026-06-20 23:21:58 on dist-test-slave-5kzz"
server_key: "e0cdfab1e68c4e8cdf1fac27ea1b930d"
server_key_iv: "8d653cedc8fc73c81ecea3a36fe19741"
server_key_version: "encryptionkey@0"
I20260620 23:21:58.355114 4701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1166c2c818374309a8c0287077db16ab"
format_stamp: "Formatted at 2026-06-20 23:21:58 on dist-test-slave-5kzz"
server_key: "e0cdfab1e68c4e8cdf1fac27ea1b930d"
server_key_iv: "8d653cedc8fc73c81ecea3a36fe19741"
server_key_version: "encryptionkey@0"
I20260620 23:21:58.359941 4701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:21:58.363323 4715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:58.364539 4701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:21:58.364711 4701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "1166c2c818374309a8c0287077db16ab"
format_stamp: "Formatted at 2026-06-20 23:21:58 on dist-test-slave-5kzz"
server_key: "e0cdfab1e68c4e8cdf1fac27ea1b930d"
server_key_iv: "8d653cedc8fc73c81ecea3a36fe19741"
server_key_version: "encryptionkey@0"
I20260620 23:21:58.364928 4701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:58.393114 4701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:58.397121 4701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:58.397348 4701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:58.398196 4701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:58.399488 4701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:58.399573 4701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:58.399649 4701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:58.399711 4701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:58.430625 4701 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45513
I20260620 23:21:58.430717 4827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45513 every 8 connection(s)
I20260620 23:21:58.431452 4701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:21:58.436810 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 4701
I20260620 23:21:58.437005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:21:58.437523 388 external_mini_cluster.cc:1428] Setting key cae7d09bcca664a6f535860dc031b927
Jun 20 23:21:58 dist-test-slave-5kzz krb5kdc[4463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:58.452993 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:58.433672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39441 (local address 127.0.97.62:41595)
0620 23:21:58.433868 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:58.433874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:58.435018 (+ 1144us) server_negotiation.cc:407] Connection header received
0620 23:21:58.436733 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:58.436739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:58.436827 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:58.436953 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:58.438871 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.439847 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.442342 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.442791 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.442883 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:58.447269 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:58.447273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:58.447277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:58.447303 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:58.449868 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:58.450767 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:58.450771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:58.450773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:58.450889 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:58.451335 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:58.451338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:58.451340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:58.451602 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:58.451701 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:58.452505 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 23:21:58.452651 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:21:58.454481 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:58.434169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41595 (local address 127.0.97.2:39441)
0620 23:21:58.434868 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:58.434910 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:58.436458 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:58.437213 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:58.437224 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:58.437957 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:58.438637 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:58.438655 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.440009 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.440014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:58.442207 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:58.442223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.442988 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.442997 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:58.443286 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:58.443981 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:58.444015 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:58.447070 (+ 3055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:58.449999 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:58.450010 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:58.450027 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:58.450625 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:58.450989 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:58.450996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:58.451003 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:58.451225 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:58.451727 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:58.451738 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:58.452139 (+ 401us) client_negotiation.cc:769] Sending connection context
0620 23:21:58.452430 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 23:21:58.452969 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":266,"threads_started":1}
I20260620 23:21:58.455873 4828 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41595
I20260620 23:21:58.456190 4828 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:58.456952 4828 heartbeater.cc:511] Master 127.0.97.62:41595 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:21:58.458380 4507 ts_manager.cc:194] Registered new tserver with Master: 1166c2c818374309a8c0287077db16ab (127.0.97.2:45513)
I20260620 23:21:58.459228 4507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:21:58.479502 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41595
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:21:58.781404 4833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:58.781700 4833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:58.781791 4833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:58.788913 4833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:58.789023 4833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:58.789130 4833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:21:58.798154 4833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:21:58.799254 4833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:21:58.800410 4833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:21:58.812441 4838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:58.812505 4839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:21:58.812623 4841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:21:58.813282 4833 server_base.cc:1034] running on GCE node
I20260620 23:21:58.813802 4833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:21:58.814584 4833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:21:58.815768 4833 hybrid_clock.cc:648] HybridClock initialized: now 1781997718815714 us; error 49 us; skew 500 ppm
I20260620 23:21:58.816017 4833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:21:58 dist-test-slave-5kzz krb5kdc[4463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:21:58.821728 4833 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:21:58.822839 4833 fs_manager.cc:359] Metadata directory not provided
I20260620 23:21:58.822923 4833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:21:58.823160 4833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:21:58.825780 4833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "b4d054efe7854afa9a65b2cb102c041e"
format_stamp: "Formatted at 2026-06-20 23:21:58 on dist-test-slave-5kzz"
server_key: "431c9cd3054861b083ec8bd1c67bb570"
server_key_iv: "7eb2ec4558f0c69c5466de150e83c34d"
server_key_version: "encryptionkey@0"
I20260620 23:21:58.826505 4833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b4d054efe7854afa9a65b2cb102c041e"
format_stamp: "Formatted at 2026-06-20 23:21:58 on dist-test-slave-5kzz"
server_key: "431c9cd3054861b083ec8bd1c67bb570"
server_key_iv: "7eb2ec4558f0c69c5466de150e83c34d"
server_key_version: "encryptionkey@0"
I20260620 23:21:58.830895 4833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:21:58.833894 4847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:58.835136 4833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:21:58.835274 4833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "b4d054efe7854afa9a65b2cb102c041e"
format_stamp: "Formatted at 2026-06-20 23:21:58 on dist-test-slave-5kzz"
server_key: "431c9cd3054861b083ec8bd1c67bb570"
server_key_iv: "7eb2ec4558f0c69c5466de150e83c34d"
server_key_version: "encryptionkey@0"
I20260620 23:21:58.835433 4833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:21:58.858186 4833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:21:58.861704 4833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:21:58.861898 4833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:21:58.862802 4833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:21:58.864033 4833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:21:58.864120 4833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:58.864225 4833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:21:58.864306 4833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:21:58.895989 4833 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37227
I20260620 23:21:58.896119 4959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37227 every 8 connection(s)
I20260620 23:21:58.896847 4833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:21:58.906026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 4833
I20260620 23:21:58.906230 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:21:58.906737 388 external_mini_cluster.cc:1428] Setting key 6936b6f92f624b9aa9c6a1fbec519f5a
Jun 20 23:21:58 dist-test-slave-5kzz krb5kdc[4463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:21:58.915633 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:58.899177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57455 (local address 127.0.97.62:41595)
0620 23:21:58.899350 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:58.899355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:58.900426 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 23:21:58.902230 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:58.902235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:58.902318 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:58.902439 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:58.903880 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.904633 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.906442 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.906933 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.907033 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:58.910503 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:58.910507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:58.910510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:58.910540 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:58.913029 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:58.913756 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:58.913760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:58.913762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:58.913874 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:58.914209 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:58.914212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:58.914214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:58.914455 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:58.914559 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:58.915165 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 23:21:58.915330 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:21:58.916567 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:58.899623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41595 (local address 127.0.97.3:57455)
0620 23:21:58.900271 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:58.900308 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:58.901976 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:58.902668 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:58.902696 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:58.903182 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:21:58.903723 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:58.903738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.904777 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.904782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:58.906250 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:58.906265 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.907150 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.907157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:58.907387 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:58.908100 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:21:58.908128 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:21:58.910314 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:21:58.913182 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:58.913188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:58.913198 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:58.913661 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:58.913962 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:21:58.913965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:21:58.913968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:21:58.914123 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:21:58.914552 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:21:58.914556 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:21:58.914881 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 23:21:58.915082 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 23:21:58.915486 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":278,"threads_started":1}
I20260620 23:21:58.917770 4960 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41595
I20260620 23:21:58.918059 4960 heartbeater.cc:464] Registering TS with master...
I20260620 23:21:58.918625 4960 heartbeater.cc:511] Master 127.0.97.62:41595 requested a full tablet report, sending...
I20260620 23:21:58.919914 4507 ts_manager.cc:194] Registered new tserver with Master: b4d054efe7854afa9a65b2cb102c041e (127.0.97.3:37227)
I20260620 23:21:58.920782 4507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57455
I20260620 23:21:58.921643 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:21:58.938341 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:58.926239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.0.97.62:41595)
0620 23:21:58.926379 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:58.926383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:58.926871 (+ 488us) server_negotiation.cc:407] Connection header received
0620 23:21:58.926987 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:58.926991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:58.927039 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:58.927111 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:58.928452 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.929050 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.930288 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.930937 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.931071 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:58.934070 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:58.934075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:58.934078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:58.934104 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:58.936299 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:58.936835 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:58.936838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:58.936841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:58.936937 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:58.937222 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:58.937225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:58.937227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:58.937482 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:58.937640 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:58.937911 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 23:21:58.938043 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:21:58.942279 4506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:21:58.948820 4506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:21:58.973269 4695 heartbeater.cc:503] Master 127.0.97.62:41595 was elected leader, sending a full tablet report...
I20260620 23:21:58.981217 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:58.959747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.0.97.2:45513)
0620 23:21:58.972130 (+ 12383us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:58.972139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:58.972928 (+ 789us) server_negotiation.cc:407] Connection header received
0620 23:21:58.973659 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:58.973668 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:58.973950 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:58.974132 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:58.975094 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.976357 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.978746 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.980078 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.980241 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:58.980437 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:58.980685 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 23:21:58.980847 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12203,"thread_start_us":12406,"threads_started":1}
I20260620 23:21:58.981284 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:58.959589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45513 (local address 127.0.0.1:56974)
0620 23:21:58.972753 (+ 13164us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:58.972799 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:58.972977 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:58.974427 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:58.974439 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:58.974466 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:58.974941 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:58.974960 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.976479 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.976487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:58.978550 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:58.978564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.980289 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.980294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:58.980460 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:58.980505 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 23:21:58.980648 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 23:21:58.980831 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12902,"mutex_wait_us":170,"thread_start_us":12852,"threads_started":1}
I20260620 23:21:58.981945 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:58.960178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35245 (local address 127.0.0.1:49566)
0620 23:21:58.973539 (+ 13361us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:58.973554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:58.973645 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:58.974487 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:58.974491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:58.974523 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:58.975597 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:58.975610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.977420 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.977425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:58.979314 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:58.979323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.981144 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.981150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:58.981331 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:58.981364 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:21:58.981434 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 23:21:58.981548 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13152,"mutex_wait_us":20,"thread_start_us":12917,"threads_started":1}
I20260620 23:21:58.982589 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:58.959832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37227 (local address 127.0.0.1:43300)
0620 23:21:58.973395 (+ 13563us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:58.973411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:58.973528 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:58.975014 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:58.975018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:58.975049 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:58.975456 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:58.975468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.977480 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.977484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:58.979800 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:58.979813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.981791 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.981797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:58.981998 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:58.982042 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 23:21:58.982130 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:21:58.982260 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13440,"mutex_wait_us":2,"thread_start_us":10461,"threads_started":1}
I20260620 23:21:58.982720 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:58.961496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.0.97.1:35245)
0620 23:21:58.972478 (+ 10982us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:58.972487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:58.973745 (+ 1258us) server_negotiation.cc:407] Connection header received
0620 23:21:58.973824 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:58.973830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:58.974050 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:58.974255 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:58.975775 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.977295 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.979450 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.980627 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.980756 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:58.980961 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:58.982227 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0620 23:21:58.982380 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10782,"thread_start_us":10624,"threads_started":1}
I20260620 23:21:58.982836 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:58.960925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.0.97.3:37227)
0620 23:21:58.972925 (+ 12000us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:58.972933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:58.973654 (+ 721us) server_negotiation.cc:407] Connection header received
0620 23:21:58.973766 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:58.973773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:58.974490 (+ 717us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:58.974671 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:58.975645 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.977281 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.979958 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:58.981338 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:58.981458 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:58.981665 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:58.982205 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 23:21:58.982362 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11837,"thread_start_us":11680,"threads_started":1}
I20260620 23:21:58.985842 4630 tablet_service.cc:1467] Processing CreateTablet for tablet 440bf23784554e388a8e0bef8f55996d (DEFAULT_TABLE table=test-table [id=22e86d6847d04e958a47ec62510de3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:58.986191 4895 tablet_service.cc:1467] Processing CreateTablet for tablet 440bf23784554e388a8e0bef8f55996d (DEFAULT_TABLE table=test-table [id=22e86d6847d04e958a47ec62510de3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:58.986186 4763 tablet_service.cc:1467] Processing CreateTablet for tablet 440bf23784554e388a8e0bef8f55996d (DEFAULT_TABLE table=test-table [id=22e86d6847d04e958a47ec62510de3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:21:58.987080 4630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440bf23784554e388a8e0bef8f55996d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:58.987672 4895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440bf23784554e388a8e0bef8f55996d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:58.987673 4763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440bf23784554e388a8e0bef8f55996d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:21:58.997490 4978 tablet_bootstrap.cc:492] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8: Bootstrap starting.
I20260620 23:21:59.000945 4978 tablet_bootstrap.cc:654] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:59.001657 4979 tablet_bootstrap.cc:492] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab: Bootstrap starting.
I20260620 23:21:59.002290 4978 log.cc:824] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:59.002362 4980 tablet_bootstrap.cc:492] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e: Bootstrap starting.
I20260620 23:21:59.005296 4979 tablet_bootstrap.cc:654] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:59.006126 4980 tablet_bootstrap.cc:654] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e: Neither blocks nor log segments found. Creating new log.
I20260620 23:21:59.006524 4979 log.cc:824] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:59.007418 4980 log.cc:824] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:21:59.008087 4978 tablet_bootstrap.cc:492] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8: No bootstrap required, opened a new log
I20260620 23:21:59.008363 4978 ts_tablet_manager.cc:1397] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 23:21:59.009641 4979 tablet_bootstrap.cc:492] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab: No bootstrap required, opened a new log
I20260620 23:21:59.009907 4979 ts_tablet_manager.cc:1397] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 23:21:59.011546 4980 tablet_bootstrap.cc:492] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e: No bootstrap required, opened a new log
I20260620 23:21:59.011781 4980 ts_tablet_manager.cc:1397] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 23:21:59.018895 4980 raft_consensus.cc:348] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1166c2c818374309a8c0287077db16ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45513 } } peers { permanent_uuid: "b4d054efe7854afa9a65b2cb102c041e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37227 } } peers { permanent_uuid: "65065b364d41402d80bcb5edd9599ee8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35245 } }
I20260620 23:21:59.018893 4978 raft_consensus.cc:348] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1166c2c818374309a8c0287077db16ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45513 } } peers { permanent_uuid: "b4d054efe7854afa9a65b2cb102c041e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37227 } } peers { permanent_uuid: "65065b364d41402d80bcb5edd9599ee8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35245 } }
I20260620 23:21:59.019289 4980 raft_consensus.cc:374] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:59.019307 4978 raft_consensus.cc:374] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:59.019388 4978 raft_consensus.cc:729] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65065b364d41402d80bcb5edd9599ee8, State: Initialized, Role: FOLLOWER
I20260620 23:21:59.019416 4980 raft_consensus.cc:729] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d054efe7854afa9a65b2cb102c041e, State: Initialized, Role: FOLLOWER
I20260620 23:21:59.019884 4979 raft_consensus.cc:348] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1166c2c818374309a8c0287077db16ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45513 } } peers { permanent_uuid: "b4d054efe7854afa9a65b2cb102c041e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37227 } } peers { permanent_uuid: "65065b364d41402d80bcb5edd9599ee8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35245 } }
I20260620 23:21:59.020067 4978 consensus_queue.cc:260] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1166c2c818374309a8c0287077db16ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45513 } } peers { permanent_uuid: "b4d054efe7854afa9a65b2cb102c041e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37227 } } peers { permanent_uuid: "65065b364d41402d80bcb5edd9599ee8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35245 } }
I20260620 23:21:59.020277 4979 raft_consensus.cc:374] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:21:59.020071 4980 consensus_queue.cc:260] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1166c2c818374309a8c0287077db16ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45513 } } peers { permanent_uuid: "b4d054efe7854afa9a65b2cb102c041e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37227 } } peers { permanent_uuid: "65065b364d41402d80bcb5edd9599ee8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35245 } }
I20260620 23:21:59.020357 4979 raft_consensus.cc:729] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1166c2c818374309a8c0287077db16ab, State: Initialized, Role: FOLLOWER
I20260620 23:21:59.021144 4960 heartbeater.cc:503] Master 127.0.97.62:41595 was elected leader, sending a full tablet report...
I20260620 23:21:59.020958 4979 consensus_queue.cc:260] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1166c2c818374309a8c0287077db16ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45513 } } peers { permanent_uuid: "b4d054efe7854afa9a65b2cb102c041e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37227 } } peers { permanent_uuid: "65065b364d41402d80bcb5edd9599ee8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35245 } }
I20260620 23:21:59.021733 4980 ts_tablet_manager.cc:1428] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20260620 23:21:59.023861 4828 heartbeater.cc:503] Master 127.0.97.62:41595 was elected leader, sending a full tablet report...
I20260620 23:21:59.024518 4979 ts_tablet_manager.cc:1428] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260620 23:21:59.026293 4978 ts_tablet_manager.cc:1428] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8: Time spent starting tablet: real 0.018s user 0.011s sys 0.003s
I20260620 23:21:59.029078 4986 raft_consensus.cc:482] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:21:59.029284 4986 raft_consensus.cc:504] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1166c2c818374309a8c0287077db16ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45513 } } peers { permanent_uuid: "b4d054efe7854afa9a65b2cb102c041e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37227 } } peers { permanent_uuid: "65065b364d41402d80bcb5edd9599ee8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35245 } }
I20260620 23:21:59.030565 4986 leader_election.cc:288] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4d054efe7854afa9a65b2cb102c041e (127.0.97.3:37227), 65065b364d41402d80bcb5edd9599ee8 (127.0.97.1:35245)
I20260620 23:21:59.035120 4985 raft_consensus.cc:482] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:21:59.035367 4985 raft_consensus.cc:504] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1166c2c818374309a8c0287077db16ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45513 } } peers { permanent_uuid: "b4d054efe7854afa9a65b2cb102c041e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37227 } } peers { permanent_uuid: "65065b364d41402d80bcb5edd9599ee8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35245 } }
I20260620 23:21:59.037952 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:59.031469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39291 (local address 127.0.97.3:37227)
0620 23:21:59.031645 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:59.031651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:59.031745 (+ 94us) server_negotiation.cc:407] Connection header received
0620 23:21:59.031953 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:59.031958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:59.032025 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:59.032104 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:59.032892 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.033634 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.035624 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.036836 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.036965 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:59.037054 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:59.037595 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 23:21:59.037715 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:21:59.038158 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:59.030941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38331 (local address 127.0.97.1:35245)
0620 23:21:59.031107 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:59.031111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:59.032123 (+ 1012us) server_negotiation.cc:407] Connection header received
0620 23:21:59.032300 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:59.032305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:59.032359 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:59.032447 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:59.033078 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.033794 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.035847 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.037124 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.037286 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:59.037375 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:59.037818 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 23:21:59.037926 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:21:59.038826 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:59.031025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35245 (local address 127.0.97.2:38331)
0620 23:21:59.032029 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:59.032043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:59.032174 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:59.032464 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:59.032468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:59.032492 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:59.032946 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:59.032957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.033925 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.033930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:59.035693 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:59.035706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.037502 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.037510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:59.037666 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:59.037694 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:21:59.038296 (+ 602us) client_negotiation.cc:240] Negotiation successful
0620 23:21:59.038441 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":29,"thread_start_us":697,"threads_started":1}
I20260620 23:21:59.039245 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:59.030921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37227 (local address 127.0.97.2:39291)
0620 23:21:59.031644 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:59.031671 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:59.031814 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:59.032260 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:59.032268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:59.032297 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:59.032726 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:59.032739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.033801 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.033807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:59.035483 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:59.035494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.037236 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.037246 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:59.037415 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:59.037455 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:21:59.038551 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0620 23:21:59.038661 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":3,"thread_start_us":388,"threads_started":1}
I20260620 23:21:59.040138 4915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440bf23784554e388a8e0bef8f55996d" candidate_uuid: "1166c2c818374309a8c0287077db16ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4d054efe7854afa9a65b2cb102c041e" is_pre_election: true
I20260620 23:21:59.040591 4915 raft_consensus.cc:2454] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1166c2c818374309a8c0287077db16ab in term 0.
I20260620 23:21:59.041145 4716 leader_election.cc:302] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1166c2c818374309a8c0287077db16ab, b4d054efe7854afa9a65b2cb102c041e; no voters:
I20260620 23:21:59.041474 4986 raft_consensus.cc:2781] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:21:59.041606 4986 raft_consensus.cc:482] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:21:59.041723 4986 raft_consensus.cc:3037] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:59.042809 4985 leader_election.cc:288] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1166c2c818374309a8c0287077db16ab (127.0.97.2:45513), b4d054efe7854afa9a65b2cb102c041e (127.0.97.3:37227)
I20260620 23:21:59.043896 4986 raft_consensus.cc:504] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1166c2c818374309a8c0287077db16ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45513 } } peers { permanent_uuid: "b4d054efe7854afa9a65b2cb102c041e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37227 } } peers { permanent_uuid: "65065b364d41402d80bcb5edd9599ee8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35245 } }
I20260620 23:21:59.044530 4986 leader_election.cc:288] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [CANDIDATE]: Term 1 election: Requested vote from peers b4d054efe7854afa9a65b2cb102c041e (127.0.97.3:37227), 65065b364d41402d80bcb5edd9599ee8 (127.0.97.1:35245)
I20260620 23:21:59.044782 4915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440bf23784554e388a8e0bef8f55996d" candidate_uuid: "1166c2c818374309a8c0287077db16ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4d054efe7854afa9a65b2cb102c041e"
I20260620 23:21:59.044950 4915 raft_consensus.cc:3037] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:59.047111 4915 raft_consensus.cc:2454] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1166c2c818374309a8c0287077db16ab in term 1.
I20260620 23:21:59.047492 4716 leader_election.cc:302] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1166c2c818374309a8c0287077db16ab, b4d054efe7854afa9a65b2cb102c041e; no voters:
I20260620 23:21:59.047729 4986 raft_consensus.cc:2781] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:21:59.050962 4986 raft_consensus.cc:686] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 1 LEADER]: Becoming Leader. State: Replica: 1166c2c818374309a8c0287077db16ab, State: Running, Role: LEADER
I20260620 23:21:59.051468 4986 consensus_queue.cc:237] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1166c2c818374309a8c0287077db16ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45513 } } peers { permanent_uuid: "b4d054efe7854afa9a65b2cb102c041e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37227 } } peers { permanent_uuid: "65065b364d41402d80bcb5edd9599ee8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35245 } }
I20260620 23:21:59.053627 4650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440bf23784554e388a8e0bef8f55996d" candidate_uuid: "1166c2c818374309a8c0287077db16ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65065b364d41402d80bcb5edd9599ee8" is_pre_election: true
I20260620 23:21:59.054061 4650 raft_consensus.cc:2454] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1166c2c818374309a8c0287077db16ab in term 0.
I20260620 23:21:59.054741 4650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440bf23784554e388a8e0bef8f55996d" candidate_uuid: "1166c2c818374309a8c0287077db16ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65065b364d41402d80bcb5edd9599ee8"
I20260620 23:21:59.054921 4650 raft_consensus.cc:3037] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:21:59.058281 4650 raft_consensus.cc:2454] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1166c2c818374309a8c0287077db16ab in term 1.
I20260620 23:21:59.060029 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:59.036972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45513 (local address 127.0.97.1:38709)
0620 23:21:59.052253 (+ 15281us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:59.052272 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:59.052508 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:59.053018 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:59.053023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:59.053046 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:59.053463 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:59.053477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.054844 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.054850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:59.057286 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:59.057302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.059092 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.059098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:59.059397 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:59.059431 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:21:59.059512 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:21:59.059616 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15154,"mutex_wait_us":75,"spinlock_wait_cycles":3328,"thread_start_us":15005,"threads_started":1}
I20260620 23:21:59.060627 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:59.037411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38709 (local address 127.0.97.2:45513)
0620 23:21:59.038050 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:59.038057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:59.052381 (+ 14324us) server_negotiation.cc:407] Connection header received
0620 23:21:59.052641 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:59.052648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:59.052743 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:59.052842 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:59.053627 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.054641 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.057471 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.058746 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.058878 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:59.058961 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:59.060266 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0620 23:21:59.060396 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494}
I20260620 23:21:59.061648 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440bf23784554e388a8e0bef8f55996d" candidate_uuid: "65065b364d41402d80bcb5edd9599ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1166c2c818374309a8c0287077db16ab" is_pre_election: true
I20260620 23:21:59.061940 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:59.037023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37227 (local address 127.0.97.1:40307)
0620 23:21:59.053009 (+ 15986us) negotiation.cc:107] Waiting for socket to connect
0620 23:21:59.053025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:21:59.053212 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:21:59.053710 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:21:59.053721 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:21:59.053759 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:21:59.054298 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:59.054313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.056414 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.056422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:59.059379 (+ 2957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:21:59.059392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.061087 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.061092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:21:59.061263 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:59.061291 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:21:59.061430 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:21:59.061561 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15875,"mutex_wait_us":37,"thread_start_us":15706,"threads_started":1}
I20260620 23:21:59.061946 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:59.037660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40307 (local address 127.0.97.3:37227)
0620 23:21:59.038183 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:59.038189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:59.053090 (+ 14901us) server_negotiation.cc:407] Connection header received
0620 23:21:59.053375 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:59.053379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:59.053448 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:59.053506 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:21:59.054508 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.056019 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.059533 (+ 3514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.060695 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.060781 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:59.060880 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:59.061496 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 23:21:59.061640 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":14001,"threads_started":1}
I20260620 23:21:59.061993 4506 catalog_manager.cc:5495] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 1166c2c818374309a8c0287077db16ab (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1166c2c818374309a8c0287077db16ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1166c2c818374309a8c0287077db16ab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4d054efe7854afa9a65b2cb102c041e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65065b364d41402d80bcb5edd9599ee8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35245 } health_report { overall_health: UNKNOWN } } }
I20260620 23:21:59.063115 4915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440bf23784554e388a8e0bef8f55996d" candidate_uuid: "65065b364d41402d80bcb5edd9599ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4d054efe7854afa9a65b2cb102c041e" is_pre_election: true
I20260620 23:21:59.063418 4915 raft_consensus.cc:2379] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65065b364d41402d80bcb5edd9599ee8 in current term 1: Already voted for candidate 1166c2c818374309a8c0287077db16ab in this term.
I20260620 23:21:59.064103 4583 leader_election.cc:302] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65065b364d41402d80bcb5edd9599ee8; no voters: 1166c2c818374309a8c0287077db16ab, b4d054efe7854afa9a65b2cb102c041e
I20260620 23:21:59.064433 4985 raft_consensus.cc:2726] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 23:21:59.078187 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:59.073628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56980 (local address 127.0.97.2:45513)
0620 23:21:59.073810 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:59.073815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:59.073833 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:21:59.073996 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:59.074000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:59.074079 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:59.074191 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:21:59.074474 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.075195 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.076455 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.076830 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.076910 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:59.077283 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:21:59.077704 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:21:59.077806 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:59.077865 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 23:21:59.077914 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:21:59.085136 4650 raft_consensus.cc:1261] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [term 1 FOLLOWER]: Refusing update from remote peer 1166c2c818374309a8c0287077db16ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:59.085175 4915 raft_consensus.cc:1261] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e [term 1 FOLLOWER]: Refusing update from remote peer 1166c2c818374309a8c0287077db16ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:21:59.085930 4992 consensus_queue.cc:1035] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "65065b364d41402d80bcb5edd9599ee8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:59.086292 4986 consensus_queue.cc:1035] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4d054efe7854afa9a65b2cb102c041e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:21:59.097602 4996 mvcc.cc:205] Tried to move back new op lower bound from 7299062657364725760 to 7299062657239429120. Current Snapshot: MvccSnapshot[applied={T|T < 7299062657364725760}]
I20260620 23:21:59.108971 4507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:59.109241 4507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:21:59.112593 4507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36df571dc2624300906cad0104d09ad3: Sending DeleteTablet for 3 replicas of tablet 440bf23784554e388a8e0bef8f55996d
I20260620 23:21:59.113633 4763 tablet_service.cc:1514] Processing DeleteTablet for tablet 440bf23784554e388a8e0bef8f55996d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56974
I20260620 23:21:59.113649 4895 tablet_service.cc:1514] Processing DeleteTablet for tablet 440bf23784554e388a8e0bef8f55996d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43300
I20260620 23:21:59.114259 4630 tablet_service.cc:1514] Processing DeleteTablet for tablet 440bf23784554e388a8e0bef8f55996d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:21:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49566
I20260620 23:21:59.114586 5006 tablet_replica.cc:330] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e: stopping tablet replica
I20260620 23:21:59.114987 5006 raft_consensus.cc:2229] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:59.115176 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 4568
I20260620 23:21:59.115507 5006 raft_consensus.cc:2258] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:59.117435 5006 ts_tablet_manager.cc:1905] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:59.126226 5006 ts_tablet_manager.cc:1918] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:21:59.126415 5006 log.cc:1196] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/440bf23784554e388a8e0bef8f55996d
I20260620 23:21:59.126983 5006 ts_tablet_manager.cc:1939] T 440bf23784554e388a8e0bef8f55996d P b4d054efe7854afa9a65b2cb102c041e: Deleting consensus metadata
I20260620 23:21:59.128630 4491 catalog_manager.cc:4841] TS b4d054efe7854afa9a65b2cb102c041e (127.0.97.3:37227): tablet 440bf23784554e388a8e0bef8f55996d (table test-table [id=22e86d6847d04e958a47ec62510de3b3]) successfully deleted
I20260620 23:21:59.131143 5007 tablet_replica.cc:330] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8: stopping tablet replica
I20260620 23:21:59.131758 5007 raft_consensus.cc:2229] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:21:59.132449 5005 tablet_replica.cc:330] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab: stopping tablet replica
I20260620 23:21:59.132943 5007 raft_consensus.cc:2258] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:21:59.134936 5005 raft_consensus.cc:2229] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:21:59.135072 5007 ts_tablet_manager.cc:1905] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:59.135614 5005 raft_consensus.cc:2258] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 23:21:59 dist-test-slave-5kzz krb5kdc[4463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:21:59.137492 5005 ts_tablet_manager.cc:1905] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:21:59.142208 5007 ts_tablet_manager.cc:1918] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:21:59.142424 5007 log.cc:1196] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/440bf23784554e388a8e0bef8f55996d
I20260620 23:21:59.143054 5007 ts_tablet_manager.cc:1939] T 440bf23784554e388a8e0bef8f55996d P 65065b364d41402d80bcb5edd9599ee8: Deleting consensus metadata
I20260620 23:21:59.144694 4492 catalog_manager.cc:4841] TS 65065b364d41402d80bcb5edd9599ee8 (127.0.97.1:35245): tablet 440bf23784554e388a8e0bef8f55996d (table test-table [id=22e86d6847d04e958a47ec62510de3b3]) successfully deleted
I20260620 23:21:59.144819 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:59.117479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49568 (local address 127.0.97.1:35245)
0620 23:21:59.117628 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:59.117634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:59.126749 (+ 9115us) server_negotiation.cc:407] Connection header received
0620 23:21:59.127031 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:59.127038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:59.127130 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:59.127221 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:59.129246 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.129970 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.131541 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.132257 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.132359 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:59.137601 (+ 5242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:59.137613 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:59.137629 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:59.137673 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:59.141505 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:59.142358 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:59.142368 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:59.142371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:59.142542 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:59.143025 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:59.143031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:59.143036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:59.143449 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:59.143730 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:59.144274 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 23:21:59.144455 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:21:59.145129 5005 ts_tablet_manager.cc:1918] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:21:59.145272 5005 log.cc:1196] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/440bf23784554e388a8e0bef8f55996d
I20260620 23:21:59.145800 5005 ts_tablet_manager.cc:1939] T 440bf23784554e388a8e0bef8f55996d P 1166c2c818374309a8c0287077db16ab: Deleting consensus metadata
I20260620 23:21:59.146080 4690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:59.147367 4493 catalog_manager.cc:4841] TS 1166c2c818374309a8c0287077db16ab (127.0.97.2:45513): tablet 440bf23784554e388a8e0bef8f55996d (table test-table [id=22e86d6847d04e958a47ec62510de3b3]) successfully deleted
I20260620 23:21:59.234369 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 4568
I20260620 23:21:59.257622 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 4701
Jun 20 23:21:59 dist-test-slave-5kzz krb5kdc[4463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:21:59.274200 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:59.258338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.0.97.2:45513)
0620 23:21:59.258530 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:59.258549 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:59.258570 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:21:59.258745 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:59.258751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:59.258842 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:59.258949 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:59.260052 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.261207 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.262495 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.263157 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.263287 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:59.267278 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:59.267294 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:59.267318 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:59.267364 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:59.271646 (+ 4282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:59.272229 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:59.272238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:59.272242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:59.272378 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:59.272694 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:59.272704 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:59.272708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:59.273160 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:59.273446 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:59.273613 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 23:21:59.273837 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:21:59.274938 4823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:59.382617 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 4701
I20260620 23:21:59.403896 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 4833
Jun 20 23:21:59 dist-test-slave-5kzz krb5kdc[4463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:21:59.417973 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:21:59.404607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.0.97.3:37227)
0620 23:21:59.404792 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:21:59.404797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:21:59.404894 (+ 97us) server_negotiation.cc:407] Connection header received
0620 23:21:59.405106 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:21:59.405111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:21:59.405191 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:21:59.405297 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:21:59.406514 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.407247 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.408466 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:21:59.408867 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:21:59.408994 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:21:59.413101 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:21:59.413115 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:21:59.413131 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:21:59.413165 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:21:59.415759 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:59.416320 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:59.416326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:59.416328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:59.416441 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:21:59.416819 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:21:59.416822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:21:59.416824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:21:59.417153 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:21:59.417363 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 23:21:59.417598 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 23:21:59.417746 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:21:59.418583 4955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:59.504493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 4833
I20260620 23:21:59.520588 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 4476
I20260620 23:21:59.521591 4537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:21:59.627611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 4476
2026-06-20T23:21:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3904 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T23:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:21:59Z Disabled control of system clock
I20260620 23:21:59.690762 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37565
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:21:59.991503 5016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:21:59.991765 5016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:21:59.991927 5016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:21:59.998823 5016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:21:59.998939 5016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:21:59.999022 5016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:21:59.999128 5016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:21:59.999207 5016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:22:00.007967 5016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:00.009161 5016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:00.010916 5016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:00.022306 5022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:00.022143 5021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:00.022429 5024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:00.023365 5016 server_base.cc:1034] running on GCE node
I20260620 23:22:00.024144 5016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:00.025506 5016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:00.026753 5016 hybrid_clock.cc:648] HybridClock initialized: now 1781997720026635 us; error 58 us; skew 500 ppm
I20260620 23:22:00.027093 5016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:00.029057 5016 webserver.cc:460] Webserver started at http://127.0.97.62:43357/ using document root <none> and password file <none>
I20260620 23:22:00.029668 5016 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:00.029776 5016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:00.030074 5016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:00.032668 5016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "0847442af6b8426fb20f826f36fe5ffd"
format_stamp: "Formatted at 2026-06-20 23:22:00 on dist-test-slave-5kzz"
server_key: "6a217283860588364ae75b6aae857214"
server_key_iv: "8e556d83da5a54dba052962d36276c21"
server_key_version: "encryptionkey@0"
I20260620 23:22:00.033447 5016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "0847442af6b8426fb20f826f36fe5ffd"
format_stamp: "Formatted at 2026-06-20 23:22:00 on dist-test-slave-5kzz"
server_key: "6a217283860588364ae75b6aae857214"
server_key_iv: "8e556d83da5a54dba052962d36276c21"
server_key_version: "encryptionkey@0"
I20260620 23:22:00.038035 5016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:22:00.041090 5030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:00.042233 5016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:22:00.042390 5016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "0847442af6b8426fb20f826f36fe5ffd"
format_stamp: "Formatted at 2026-06-20 23:22:00 on dist-test-slave-5kzz"
server_key: "6a217283860588364ae75b6aae857214"
server_key_iv: "8e556d83da5a54dba052962d36276c21"
server_key_version: "encryptionkey@0"
I20260620 23:22:00.042584 5016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:00.047806 5016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:00.048466 5016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:00.048794 5016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:00.065349 5016 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37565
I20260620 23:22:00.065359 5081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37565 every 8 connection(s)
I20260620 23:22:00.066563 5016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:00.070014 5082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:00.075877 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 5016
I20260620 23:22:00.076046 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:22:00.076668 388 external_mini_cluster.cc:1428] Setting key 400b58a9ac2fa21c60cd714084af583e
I20260620 23:22:00.080047 5082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd: Bootstrap starting.
I20260620 23:22:00.083292 5082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:00.084350 5082 log.cc:824] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:00.084554 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:00.078579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.0.97.62:37565)
0620 23:22:00.079548 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:00.079558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:00.079597 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:22:00.081221 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:00.081241 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:00.081617 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:00.081919 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:00.082977 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:00.082983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:00.082995 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:00.083008 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:00.083061 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:00.083076 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:00.083220 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:00.083403 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 23:22:00.083683 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":338,"threads_started":1}
I20260620 23:22:00.087348 5082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd: No bootstrap required, opened a new log
I20260620 23:22:00.094133 5082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0847442af6b8426fb20f826f36fe5ffd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37565 } }
I20260620 23:22:00.094458 5082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:00.094569 5082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0847442af6b8426fb20f826f36fe5ffd, State: Initialized, Role: FOLLOWER
I20260620 23:22:00.095183 5082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0847442af6b8426fb20f826f36fe5ffd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37565 } }
I20260620 23:22:00.095470 5082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:00.095604 5082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:00.095721 5082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:00.097610 5082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0847442af6b8426fb20f826f36fe5ffd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37565 } }
I20260620 23:22:00.097993 5082 leader_election.cc:302] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0847442af6b8426fb20f826f36fe5ffd; no voters:
I20260620 23:22:00.098450 5082 leader_election.cc:288] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:22:00.098539 5087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:00.099223 5087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [term 1 LEADER]: Becoming Leader. State: Replica: 0847442af6b8426fb20f826f36fe5ffd, State: Running, Role: LEADER
I20260620 23:22:00.099787 5087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0847442af6b8426fb20f826f36fe5ffd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37565 } }
I20260620 23:22:00.100415 5082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:00.102425 5088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0847442af6b8426fb20f826f36fe5ffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0847442af6b8426fb20f826f36fe5ffd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37565 } } }
I20260620 23:22:00.102478 5089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0847442af6b8426fb20f826f36fe5ffd. Latest consensus state: current_term: 1 leader_uuid: "0847442af6b8426fb20f826f36fe5ffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0847442af6b8426fb20f826f36fe5ffd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37565 } } }
I20260620 23:22:00.102859 5088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:00.102905 5089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:00.105505 5094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:00.109853 5094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:00.118239 5094 catalog_manager.cc:1347] Generated new cluster ID: 6e3fa87391204e81ab66759721cf0c40
I20260620 23:22:00.118357 5094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:00.137950 5094 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:22:00.138383 5094 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:00.143632 5094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd: Generated new TSK 0
I20260620 23:22:00.144520 5094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:22:00.153501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37565
--builtin_ntp_servers=127.0.97.20:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:22:00.461992 5106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:00.462235 5106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:00.462323 5106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:00.469355 5106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:00.469442 5106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:00.469550 5106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:22:00.478407 5106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:00.479799 5106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:00.481474 5106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:00.494374 5112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:00.494413 5111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:00.494655 5114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:00.495249 5106 server_base.cc:1034] running on GCE node
I20260620 23:22:00.495842 5106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:00.496709 5106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:00.497927 5106 hybrid_clock.cc:648] HybridClock initialized: now 1781997720497892 us; error 55 us; skew 500 ppm
I20260620 23:22:00.498222 5106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:00.500468 5106 webserver.cc:460] Webserver started at http://127.0.97.1:34745/ using document root <none> and password file <none>
I20260620 23:22:00.501122 5106 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:00.501250 5106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:00.501551 5106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:00.504218 5106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "cee28583cddb42faad56ade0064e6573"
format_stamp: "Formatted at 2026-06-20 23:22:00 on dist-test-slave-5kzz"
server_key: "978d5d9b03a6dbab6afd6d2b7d2b96f9"
server_key_iv: "978dc3028a4e272f0c0d583ab1723aa6"
server_key_version: "encryptionkey@0"
I20260620 23:22:00.504899 5106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cee28583cddb42faad56ade0064e6573"
format_stamp: "Formatted at 2026-06-20 23:22:00 on dist-test-slave-5kzz"
server_key: "978d5d9b03a6dbab6afd6d2b7d2b96f9"
server_key_iv: "978dc3028a4e272f0c0d583ab1723aa6"
server_key_version: "encryptionkey@0"
I20260620 23:22:00.509434 5106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260620 23:22:00.512527 5120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:00.513747 5106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:22:00.513911 5106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "cee28583cddb42faad56ade0064e6573"
format_stamp: "Formatted at 2026-06-20 23:22:00 on dist-test-slave-5kzz"
server_key: "978d5d9b03a6dbab6afd6d2b7d2b96f9"
server_key_iv: "978dc3028a4e272f0c0d583ab1723aa6"
server_key_version: "encryptionkey@0"
I20260620 23:22:00.514137 5106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:00.519373 5106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:00.520099 5106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:00.520364 5106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:00.521212 5106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:00.522511 5106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:00.522630 5106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:00.522779 5106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:00.522841 5106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:00.553956 5106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35929
I20260620 23:22:00.554003 5232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35929 every 8 connection(s)
I20260620 23:22:00.555358 5106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:22:00.560289 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 5106
I20260620 23:22:00.560467 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:22:00.561064 388 external_mini_cluster.cc:1428] Setting key bda777b1298cf18140d747015701bcd3
I20260620 23:22:00.563490 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:00.557897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51159 (local address 127.0.97.62:37565)
0620 23:22:00.558088 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:00.558094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:00.559491 (+ 1397us) server_negotiation.cc:407] Connection header received
0620 23:22:00.561274 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:00.561280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:00.561348 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:00.561463 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:00.562789 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:00.562795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:00.562799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:00.562823 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:00.562849 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:00.562860 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:00.562982 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:00.563184 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 23:22:00.563247 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:22:00.563799 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37565
--builtin_ntp_servers=127.0.97.20:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 23:22:00.564483 5235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:00.558449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37565 (local address 127.0.97.1:51159)
0620 23:22:00.559350 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:00.559385 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:00.560984 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:00.561680 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:00.561688 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:00.561778 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:00.562421 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:00.562424 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:00.562471 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:00.562575 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:00.562951 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:00.562962 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:00.562971 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:22:00.563276 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 23:22:00.563521 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":313,"threads_started":1}
I20260620 23:22:00.565860 5233 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37565
I20260620 23:22:00.566140 5233 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:00.566720 5233 heartbeater.cc:511] Master 127.0.97.62:37565 requested a full tablet report, sending...
I20260620 23:22:00.568706 5047 ts_manager.cc:194] Registered new tserver with Master: cee28583cddb42faad56ade0064e6573 (127.0.97.1:35929)
W20260620 23:22:00.868916 5236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:00.869165 5236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:00.869335 5236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:00.876269 5236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:00.876353 5236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:00.876543 5236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:22:00.885202 5236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:00.886372 5236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:00.888023 5236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:00.900668 5244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:00.900763 5241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:00.900736 5242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:00.901507 5236 server_base.cc:1034] running on GCE node
I20260620 23:22:00.902108 5236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:00.903034 5236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:00.904246 5236 hybrid_clock.cc:648] HybridClock initialized: now 1781997720904200 us; error 56 us; skew 500 ppm
I20260620 23:22:00.904596 5236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:00.906755 5236 webserver.cc:460] Webserver started at http://127.0.97.2:41907/ using document root <none> and password file <none>
I20260620 23:22:00.907389 5236 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:00.907510 5236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:00.907828 5236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:00.910468 5236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "759db7158de74db5b84632e6987375f6"
format_stamp: "Formatted at 2026-06-20 23:22:00 on dist-test-slave-5kzz"
server_key: "342606de3af1993cdb30d57dfdcd324d"
server_key_iv: "bf3e48b65cac4115d579490a6f976232"
server_key_version: "encryptionkey@0"
I20260620 23:22:00.911266 5236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "759db7158de74db5b84632e6987375f6"
format_stamp: "Formatted at 2026-06-20 23:22:00 on dist-test-slave-5kzz"
server_key: "342606de3af1993cdb30d57dfdcd324d"
server_key_iv: "bf3e48b65cac4115d579490a6f976232"
server_key_version: "encryptionkey@0"
I20260620 23:22:00.915851 5236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:22:00.919203 5250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:00.920367 5236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:22:00.920531 5236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "759db7158de74db5b84632e6987375f6"
format_stamp: "Formatted at 2026-06-20 23:22:00 on dist-test-slave-5kzz"
server_key: "342606de3af1993cdb30d57dfdcd324d"
server_key_iv: "bf3e48b65cac4115d579490a6f976232"
server_key_version: "encryptionkey@0"
I20260620 23:22:00.920742 5236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:00.925804 5236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:00.926501 5236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:00.926826 5236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:00.927632 5236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:00.928921 5236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:00.929036 5236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:00.929157 5236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:00.929247 5236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:00.961777 5236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38417
I20260620 23:22:00.961859 5362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38417 every 8 connection(s)
I20260620 23:22:00.963132 5236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:22:00.970422 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:00.965506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36505 (local address 127.0.97.62:37565)
0620 23:22:00.965718 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:00.965725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:00.967079 (+ 1354us) server_negotiation.cc:407] Connection header received
0620 23:22:00.968610 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:00.968614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:00.968662 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:00.968750 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:00.969843 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:00.969847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:00.969850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:00.969865 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:00.969894 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:00.969904 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:00.970002 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:00.970139 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:22:00.970188 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:22:00.971262 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:00.966030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37565 (local address 127.0.97.2:36505)
0620 23:22:00.966939 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:00.966979 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:00.968379 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:00.968930 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:00.968937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:00.969016 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:00.969590 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:00.969592 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:00.969626 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:00.969699 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:00.969984 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:00.969990 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:00.969995 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:22:00.970179 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 23:22:00.970374 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":264,"threads_started":1}
I20260620 23:22:00.971799 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 5236
I20260620 23:22:00.971954 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:22:00.972383 388 external_mini_cluster.cc:1428] Setting key 1e0c2cf410dbb316f11aff57d7e71867
I20260620 23:22:00.972590 5363 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37565
I20260620 23:22:00.972893 5363 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:00.973546 5363 heartbeater.cc:511] Master 127.0.97.62:37565 requested a full tablet report, sending...
I20260620 23:22:00.975034 5047 ts_manager.cc:194] Registered new tserver with Master: 759db7158de74db5b84632e6987375f6 (127.0.97.2:38417)
I20260620 23:22:00.975250 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37565
--builtin_ntp_servers=127.0.97.20:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:22:01.283860 5366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:01.284159 5366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:01.284322 5366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:01.291328 5366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:01.291447 5366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:01.291668 5366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:22:01.300191 5366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:01.301476 5366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:01.303297 5366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:01.315340 5371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:01.315409 5372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:01.315377 5374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:01.316313 5366 server_base.cc:1034] running on GCE node
I20260620 23:22:01.316879 5366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:01.317816 5366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:01.319015 5366 hybrid_clock.cc:648] HybridClock initialized: now 1781997721318956 us; error 73 us; skew 500 ppm
I20260620 23:22:01.319310 5366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:01.321425 5366 webserver.cc:460] Webserver started at http://127.0.97.3:41325/ using document root <none> and password file <none>
I20260620 23:22:01.322065 5366 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:01.322183 5366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:01.322468 5366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:01.325107 5366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "d443c357eefe44198d3d0425172aed54"
format_stamp: "Formatted at 2026-06-20 23:22:01 on dist-test-slave-5kzz"
server_key: "c15ceaa49eb2fe52c972edf00188172d"
server_key_iv: "c8942f89aad99bf23644254146a701d1"
server_key_version: "encryptionkey@0"
I20260620 23:22:01.325874 5366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d443c357eefe44198d3d0425172aed54"
format_stamp: "Formatted at 2026-06-20 23:22:01 on dist-test-slave-5kzz"
server_key: "c15ceaa49eb2fe52c972edf00188172d"
server_key_iv: "c8942f89aad99bf23644254146a701d1"
server_key_version: "encryptionkey@0"
I20260620 23:22:01.330641 5366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 23:22:01.333870 5380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:01.335188 5366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:22:01.335350 5366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "d443c357eefe44198d3d0425172aed54"
format_stamp: "Formatted at 2026-06-20 23:22:01 on dist-test-slave-5kzz"
server_key: "c15ceaa49eb2fe52c972edf00188172d"
server_key_iv: "c8942f89aad99bf23644254146a701d1"
server_key_version: "encryptionkey@0"
I20260620 23:22:01.335595 5366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:01.340766 5366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:01.341549 5366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:01.341804 5366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:01.342630 5366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:01.343935 5366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:01.344060 5366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:01.344184 5366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:01.344269 5366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:01.375828 5366 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33941
I20260620 23:22:01.375874 5492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33941 every 8 connection(s)
I20260620 23:22:01.377216 5366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:22:01.382134 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 5366
I20260620 23:22:01.382292 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:22:01.382836 388 external_mini_cluster.cc:1428] Setting key eb76c08eb498d478e358c7da2ba23d07
I20260620 23:22:01.386034 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.379774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51075 (local address 127.0.97.62:37565)
0620 23:22:01.379954 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.379958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.381338 (+ 1380us) server_negotiation.cc:407] Connection header received
0620 23:22:01.383660 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.383665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.383720 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.383801 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.385313 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.385318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.385322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.385339 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.385374 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.385388 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.385500 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.385687 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.385745 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:22:01.386900 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.380290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37565 (local address 127.0.97.3:51075)
0620 23:22:01.381182 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.381232 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:01.383360 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:01.384057 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:01.384068 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:01.384186 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:01.384996 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:01.384999 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:01.385046 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:01.385138 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:01.385483 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:01.385490 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:01.385495 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:22:01.385710 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 23:22:01.385954 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":254,"threads_started":1}
I20260620 23:22:01.388041 5493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37565
I20260620 23:22:01.388265 5493 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:01.388770 5493 heartbeater.cc:511] Master 127.0.97.62:37565 requested a full tablet report, sending...
I20260620 23:22:01.389927 5047 ts_manager.cc:194] Registered new tserver with Master: d443c357eefe44198d3d0425172aed54 (127.0.97.3:33941)
I20260620 23:22:01.397977 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:22:01.404513 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.402154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.0.97.62:37565)
0620 23:22:01.402351 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.402359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.402839 (+ 480us) server_negotiation.cc:407] Connection header received
0620 23:22:01.402962 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.402966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.403013 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.403089 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.403941 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.403944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.403948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.403961 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.403988 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.403998 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.404113 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.404156 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.404208 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:22:01.408048 5047 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:22:01.415390 5047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:22:01.432255 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.427109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.0.97.1:35929)
0620 23:22:01.428046 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.428056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.429892 (+ 1836us) server_negotiation.cc:407] Connection header received
0620 23:22:01.430053 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.430061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.430258 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.430384 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.431475 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.431481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.431490 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.431505 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.431552 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.431587 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.431699 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.431883 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.431975 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":1166,"threads_started":1}
I20260620 23:22:01.432305 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.426420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33941 (local address 127.0.0.1:43704)
0620 23:22:01.427468 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.427503 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:01.427624 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:01.429558 (+ 1934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:01.429565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:01.429642 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:01.430461 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:01.430464 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:01.430489 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:01.430561 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:01.431625 (+ 1064us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:01.431638 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:01.431651 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:22:01.431802 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 23:22:01.431884 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"spinlock_wait_cycles":768,"thread_start_us":771,"threads_started":1}
I20260620 23:22:01.433127 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.426635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.0.97.3:33941)
0620 23:22:01.428902 (+ 2267us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.428911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.428965 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:22:01.429032 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.429042 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.429342 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.430086 (+ 744us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.431383 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.431388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.431420 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.431435 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.431480 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.431498 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.432583 (+ 1085us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.432653 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.432744 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2088,"thread_start_us":1880,"threads_started":1}
I20260620 23:22:01.433375 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.426831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35929 (local address 127.0.0.1:36944)
0620 23:22:01.429743 (+ 2912us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.429768 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:01.429901 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:01.430442 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:01.430448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:01.430477 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:01.431255 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:01.431260 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:01.431279 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:01.431331 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:01.431740 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:01.431744 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:01.431748 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:22:01.432930 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0620 23:22:01.432979 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2716,"thread_start_us":4731,"threads_started":1}
I20260620 23:22:01.436510 5168 tablet_service.cc:1467] Processing CreateTablet for tablet ebc52ef0a3014697bfead407187dbd04 (DEFAULT_TABLE table=test-table [id=73f4b01c44bf476d869d0af0166cc3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:01.436484 5428 tablet_service.cc:1467] Processing CreateTablet for tablet ebc52ef0a3014697bfead407187dbd04 (DEFAULT_TABLE table=test-table [id=73f4b01c44bf476d869d0af0166cc3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:01.437618 5428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc52ef0a3014697bfead407187dbd04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:01.437620 5168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc52ef0a3014697bfead407187dbd04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:01.443888 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.427985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38417 (local address 127.0.0.1:36814)
0620 23:22:01.431799 (+ 3814us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.431813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:01.431961 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:01.434008 (+ 2047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:01.434011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:01.434027 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:01.442452 (+ 8425us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:01.442456 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:01.442472 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:01.442804 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:01.443311 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:01.443315 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:01.443319 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:22:01.443421 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:22:01.443467 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3713,"thread_start_us":1410,"threads_started":1}
I20260620 23:22:01.443918 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.428465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.0.97.2:38417)
0620 23:22:01.429203 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.429215 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.431893 (+ 2678us) server_negotiation.cc:407] Connection header received
0620 23:22:01.432096 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.432104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.432291 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.432433 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.442976 (+ 10543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.442986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.443000 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.443020 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.443086 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.443105 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.443229 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.443473 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.443582 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":396,"threads_started":1}
I20260620 23:22:01.447903 5298 tablet_service.cc:1467] Processing CreateTablet for tablet ebc52ef0a3014697bfead407187dbd04 (DEFAULT_TABLE table=test-table [id=73f4b01c44bf476d869d0af0166cc3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:01.449018 5511 tablet_bootstrap.cc:492] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573: Bootstrap starting.
I20260620 23:22:01.449870 5298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc52ef0a3014697bfead407187dbd04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:01.451311 5512 tablet_bootstrap.cc:492] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54: Bootstrap starting.
I20260620 23:22:01.452618 5511 tablet_bootstrap.cc:654] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:01.454439 5511 log.cc:824] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:01.454775 5512 tablet_bootstrap.cc:654] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:01.455967 5512 log.cc:824] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:01.457391 5511 tablet_bootstrap.cc:492] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573: No bootstrap required, opened a new log
I20260620 23:22:01.457652 5511 ts_tablet_manager.cc:1397] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:22:01.458846 5512 tablet_bootstrap.cc:492] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54: No bootstrap required, opened a new log
I20260620 23:22:01.459120 5512 ts_tablet_manager.cc:1397] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 23:22:01.462054 5515 tablet_bootstrap.cc:492] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6: Bootstrap starting.
I20260620 23:22:01.464823 5515 tablet_bootstrap.cc:654] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:01.465731 5515 log.cc:824] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:01.467777 5511 raft_consensus.cc:348] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 } } peers { permanent_uuid: "759db7158de74db5b84632e6987375f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } } peers { permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 } }
I20260620 23:22:01.468216 5511 raft_consensus.cc:374] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:01.468312 5511 raft_consensus.cc:729] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee28583cddb42faad56ade0064e6573, State: Initialized, Role: FOLLOWER
I20260620 23:22:01.468464 5515 tablet_bootstrap.cc:492] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6: No bootstrap required, opened a new log
I20260620 23:22:01.468750 5515 ts_tablet_manager.cc:1397] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 23:22:01.469019 5512 raft_consensus.cc:348] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 } } peers { permanent_uuid: "759db7158de74db5b84632e6987375f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } } peers { permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 } }
I20260620 23:22:01.469174 5511 consensus_queue.cc:260] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 } } peers { permanent_uuid: "759db7158de74db5b84632e6987375f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } } peers { permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 } }
I20260620 23:22:01.469431 5512 raft_consensus.cc:374] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:01.469548 5512 raft_consensus.cc:729] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d443c357eefe44198d3d0425172aed54, State: Initialized, Role: FOLLOWER
I20260620 23:22:01.470229 5512 consensus_queue.cc:260] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 } } peers { permanent_uuid: "759db7158de74db5b84632e6987375f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } } peers { permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 } }
I20260620 23:22:01.471364 5233 heartbeater.cc:503] Master 127.0.97.62:37565 was elected leader, sending a full tablet report...
I20260620 23:22:01.473886 5511 ts_tablet_manager.cc:1428] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260620 23:22:01.475358 5493 heartbeater.cc:503] Master 127.0.97.62:37565 was elected leader, sending a full tablet report...
I20260620 23:22:01.475973 5512 ts_tablet_manager.cc:1428] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
I20260620 23:22:01.477831 5515 raft_consensus.cc:348] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 } } peers { permanent_uuid: "759db7158de74db5b84632e6987375f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } } peers { permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 } }
I20260620 23:22:01.478168 5515 raft_consensus.cc:374] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:01.478271 5515 raft_consensus.cc:729] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759db7158de74db5b84632e6987375f6, State: Initialized, Role: FOLLOWER
I20260620 23:22:01.478883 5515 consensus_queue.cc:260] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 } } peers { permanent_uuid: "759db7158de74db5b84632e6987375f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } } peers { permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 } }
I20260620 23:22:01.479815 5363 heartbeater.cc:503] Master 127.0.97.62:37565 was elected leader, sending a full tablet report...
I20260620 23:22:01.480410 5515 ts_tablet_manager.cc:1428] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260620 23:22:01.532323 5519 raft_consensus.cc:482] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:01.532548 5519 raft_consensus.cc:504] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 } } peers { permanent_uuid: "759db7158de74db5b84632e6987375f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } } peers { permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 } }
I20260620 23:22:01.533438 5517 raft_consensus.cc:482] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:01.533722 5517 raft_consensus.cc:504] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 } } peers { permanent_uuid: "759db7158de74db5b84632e6987375f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } } peers { permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 } }
I20260620 23:22:01.534255 5519 leader_election.cc:288] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d443c357eefe44198d3d0425172aed54 (127.0.97.3:33941), cee28583cddb42faad56ade0064e6573 (127.0.97.1:35929)
I20260620 23:22:01.535712 5517 leader_election.cc:288] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d443c357eefe44198d3d0425172aed54 (127.0.97.3:33941), 759db7158de74db5b84632e6987375f6 (127.0.97.2:38417)
I20260620 23:22:01.539285 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.535808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57345 (local address 127.0.97.3:33941)
0620 23:22:01.535962 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.535970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.536481 (+ 511us) server_negotiation.cc:407] Connection header received
0620 23:22:01.536743 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.536748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.536808 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.536886 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.538643 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.538649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.538657 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.538700 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.538739 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.538758 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.538875 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.538963 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.539011 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:22:01.539736 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.534602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35929 (local address 127.0.97.2:55705)
0620 23:22:01.537319 (+ 2717us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.537338 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:01.537464 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:01.537866 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:01.537869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:01.537890 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:01.538551 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:01.538555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:01.538598 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:01.538648 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:01.539030 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:01.539034 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:01.539039 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:22:01.539200 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 23:22:01.539256 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2587,"thread_start_us":816,"threads_started":1}
I20260620 23:22:01.539974 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.534472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33941 (local address 127.0.97.2:57345)
0620 23:22:01.536375 (+ 1903us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.536401 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:01.536573 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:01.537097 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:01.537101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:01.537129 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:01.538064 (+ 935us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:01.538070 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:01.538097 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:01.538161 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:01.538847 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:01.538852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:01.538859 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:22:01.538954 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 23:22:01.539063 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1722,"thread_start_us":1508,"threads_started":1}
I20260620 23:22:01.540057 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.534984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55705 (local address 127.0.97.1:35929)
0620 23:22:01.535215 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.535224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.537410 (+ 2186us) server_negotiation.cc:407] Connection header received
0620 23:22:01.537578 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.537583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.537632 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.537714 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.538823 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.538831 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.538854 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.538870 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.538904 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.538918 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.539052 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.539710 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.539764 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:22:01.540843 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.535783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33941 (local address 127.0.97.1:46767)
0620 23:22:01.536782 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.536805 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:01.536940 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:01.538333 (+ 1393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:01.538338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:01.538365 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:01.539595 (+ 1230us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:01.539600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:01.539617 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:01.539661 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:01.540262 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:01.540268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:01.540273 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:22:01.540386 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:22:01.540421 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":604,"threads_started":1}
I20260620 23:22:01.541025 5448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc52ef0a3014697bfead407187dbd04" candidate_uuid: "759db7158de74db5b84632e6987375f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d443c357eefe44198d3d0425172aed54" is_pre_election: true
I20260620 23:22:01.541055 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc52ef0a3014697bfead407187dbd04" candidate_uuid: "759db7158de74db5b84632e6987375f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee28583cddb42faad56ade0064e6573" is_pre_election: true
I20260620 23:22:01.541651 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.536123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46767 (local address 127.0.97.3:33941)
0620 23:22:01.538011 (+ 1888us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.538019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.538040 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:22:01.538099 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.538106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.538157 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.538531 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.540045 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.540049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.540054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.540070 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.540112 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.540126 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.541124 (+ 998us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.541180 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.541233 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1749,"thread_start_us":1577,"threads_started":1}
I20260620 23:22:01.541572 5448 raft_consensus.cc:2454] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759db7158de74db5b84632e6987375f6 in term 0.
I20260620 23:22:01.541785 5188 raft_consensus.cc:2454] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759db7158de74db5b84632e6987375f6 in term 0.
I20260620 23:22:01.542330 5448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc52ef0a3014697bfead407187dbd04" candidate_uuid: "cee28583cddb42faad56ade0064e6573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d443c357eefe44198d3d0425172aed54" is_pre_election: true
I20260620 23:22:01.542368 5251 leader_election.cc:302] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759db7158de74db5b84632e6987375f6, d443c357eefe44198d3d0425172aed54; no voters:
I20260620 23:22:01.542513 5448 raft_consensus.cc:2454] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee28583cddb42faad56ade0064e6573 in term 0.
I20260620 23:22:01.542815 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.537089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58031 (local address 127.0.97.2:38417)
0620 23:22:01.537230 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.537239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.539978 (+ 2739us) server_negotiation.cc:407] Connection header received
0620 23:22:01.540763 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.540770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.540822 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.540885 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.542114 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.542119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.542125 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.542146 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.542176 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.542203 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.542288 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.542524 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.542572 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:22:01.542815 5519 raft_consensus.cc:2781] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:22:01.542976 5519 raft_consensus.cc:482] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:22:01.543073 5519 raft_consensus.cc:3037] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:01.543176 5121 leader_election.cc:302] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cee28583cddb42faad56ade0064e6573, d443c357eefe44198d3d0425172aed54; no voters:
I20260620 23:22:01.543545 5517 raft_consensus.cc:2781] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:22:01.543609 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.536798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38417 (local address 127.0.97.1:58031)
0620 23:22:01.539894 (+ 3096us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.539911 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:01.540610 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:01.541065 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:01.541071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:01.541095 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:01.541896 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:01.541905 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:01.541921 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:01.541959 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:01.542373 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:01.542378 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:01.542382 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:22:01.543087 (+ 705us) client_negotiation.cc:240] Negotiation successful
0620 23:22:01.543137 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2967,"thread_start_us":1538,"threads_started":1}
I20260620 23:22:01.543697 5517 raft_consensus.cc:482] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:22:01.543793 5517 raft_consensus.cc:3037] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:01.544553 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc52ef0a3014697bfead407187dbd04" candidate_uuid: "cee28583cddb42faad56ade0064e6573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759db7158de74db5b84632e6987375f6" is_pre_election: true
I20260620 23:22:01.545818 5519 raft_consensus.cc:504] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 } } peers { permanent_uuid: "759db7158de74db5b84632e6987375f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } } peers { permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 } }
I20260620 23:22:01.546277 5318 raft_consensus.cc:2379] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cee28583cddb42faad56ade0064e6573 in current term 1: Already voted for candidate 759db7158de74db5b84632e6987375f6 in this term.
I20260620 23:22:01.546294 5517 raft_consensus.cc:504] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 } } peers { permanent_uuid: "759db7158de74db5b84632e6987375f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } } peers { permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 } }
I20260620 23:22:01.546763 5519 leader_election.cc:288] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [CANDIDATE]: Term 1 election: Requested vote from peers d443c357eefe44198d3d0425172aed54 (127.0.97.3:33941), cee28583cddb42faad56ade0064e6573 (127.0.97.1:35929)
I20260620 23:22:01.546891 5517 leader_election.cc:288] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [CANDIDATE]: Term 1 election: Requested vote from peers d443c357eefe44198d3d0425172aed54 (127.0.97.3:33941), 759db7158de74db5b84632e6987375f6 (127.0.97.2:38417)
I20260620 23:22:01.546916 5448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc52ef0a3014697bfead407187dbd04" candidate_uuid: "759db7158de74db5b84632e6987375f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d443c357eefe44198d3d0425172aed54"
I20260620 23:22:01.547087 5448 raft_consensus.cc:3037] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:01.547698 5317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc52ef0a3014697bfead407187dbd04" candidate_uuid: "cee28583cddb42faad56ade0064e6573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759db7158de74db5b84632e6987375f6"
I20260620 23:22:01.547837 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc52ef0a3014697bfead407187dbd04" candidate_uuid: "759db7158de74db5b84632e6987375f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee28583cddb42faad56ade0064e6573"
I20260620 23:22:01.547987 5317 raft_consensus.cc:2379] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cee28583cddb42faad56ade0064e6573 in current term 1: Already voted for candidate 759db7158de74db5b84632e6987375f6 in this term.
I20260620 23:22:01.548075 5188 raft_consensus.cc:2379] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 759db7158de74db5b84632e6987375f6 in current term 1: Already voted for candidate cee28583cddb42faad56ade0064e6573 in this term.
I20260620 23:22:01.548604 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc52ef0a3014697bfead407187dbd04" candidate_uuid: "cee28583cddb42faad56ade0064e6573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d443c357eefe44198d3d0425172aed54"
I20260620 23:22:01.549266 5121 leader_election.cc:302] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cee28583cddb42faad56ade0064e6573; no voters: 759db7158de74db5b84632e6987375f6, d443c357eefe44198d3d0425172aed54
I20260620 23:22:01.549499 5517 raft_consensus.cc:2726] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 23:22:01.550045 5448 raft_consensus.cc:2454] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759db7158de74db5b84632e6987375f6 in term 1.
I20260620 23:22:01.550384 5251 leader_election.cc:302] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 759db7158de74db5b84632e6987375f6, d443c357eefe44198d3d0425172aed54; no voters: cee28583cddb42faad56ade0064e6573
I20260620 23:22:01.550707 5519 raft_consensus.cc:2781] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:01.551187 5519 raft_consensus.cc:686] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 1 LEADER]: Becoming Leader. State: Replica: 759db7158de74db5b84632e6987375f6, State: Running, Role: LEADER
I20260620 23:22:01.551648 5519 consensus_queue.cc:237] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 } } peers { permanent_uuid: "759db7158de74db5b84632e6987375f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } } peers { permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 } }
I20260620 23:22:01.556026 5047 catalog_manager.cc:5495] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759db7158de74db5b84632e6987375f6 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "759db7158de74db5b84632e6987375f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "759db7158de74db5b84632e6987375f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 } health_report { overall_health: UNKNOWN } } }
W20260620 23:22:01.557341 5234 tablet.cc:2369] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:22:01.588907 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.587072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.0.97.2:38417)
0620 23:22:01.587298 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.587302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.587320 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:22:01.587416 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.587419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.587507 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.587583 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.588321 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.588325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.588328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.588341 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.588368 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.588379 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.588474 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.588552 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.588593 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 23:22:01.596763 5188 raft_consensus.cc:1261] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 1 FOLLOWER]: Refusing update from remote peer 759db7158de74db5b84632e6987375f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:01.596742 5448 raft_consensus.cc:1261] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54 [term 1 FOLLOWER]: Refusing update from remote peer 759db7158de74db5b84632e6987375f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:01.597801 5519 consensus_queue.cc:1035] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cee28583cddb42faad56ade0064e6573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:01.598091 5525 consensus_queue.cc:1035] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d443c357eefe44198d3d0425172aed54" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:01.615411 5529 mvcc.cc:205] Tried to move back new op lower bound from 7299062667650048000 to 7299062667479076864. Current Snapshot: MvccSnapshot[applied={T|T < 7299062667650048000 or (T in {7299062667650048000})}]
I20260620 23:22:01.622243 5047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:01.622507 5047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:01.625844 5047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0847442af6b8426fb20f826f36fe5ffd: Sending DeleteTablet for 3 replicas of tablet ebc52ef0a3014697bfead407187dbd04
I20260620 23:22:01.626893 5428 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc52ef0a3014697bfead407187dbd04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:01 UTC) from {username='slave'} at 127.0.0.1:43704
I20260620 23:22:01.626981 5298 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc52ef0a3014697bfead407187dbd04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:01 UTC) from {username='slave'} at 127.0.0.1:36814
I20260620 23:22:01.627220 5168 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc52ef0a3014697bfead407187dbd04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:01 UTC) from {username='slave'} at 127.0.0.1:36944
I20260620 23:22:01.631067 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 5106
I20260620 23:22:01.637779 5538 tablet_replica.cc:330] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54: stopping tablet replica
I20260620 23:22:01.637985 5539 tablet_replica.cc:330] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573: stopping tablet replica
I20260620 23:22:01.638432 5539 raft_consensus.cc:2229] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:01.639109 5539 raft_consensus.cc:2258] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:01.638195 5538 raft_consensus.cc:2229] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:01.639787 5538 raft_consensus.cc:2258] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:01.641754 5539 ts_tablet_manager.cc:1905] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:01.644021 5537 tablet_replica.cc:330] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6: stopping tablet replica
I20260620 23:22:01.644488 5537 raft_consensus.cc:2229] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:22:01.645102 5537 raft_consensus.cc:2258] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:01.647192 5537 ts_tablet_manager.cc:1905] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:01.649012 5539 ts_tablet_manager.cc:1918] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:01.650615 5538 ts_tablet_manager.cc:1905] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:01.651180 5539 log.cc:1196] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/ebc52ef0a3014697bfead407187dbd04
I20260620 23:22:01.653697 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.632783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.0.97.1:35929)
0620 23:22:01.632999 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.633008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.651147 (+ 18139us) server_negotiation.cc:407] Connection header received
0620 23:22:01.651370 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.651375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.651485 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.651581 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.652740 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.652746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.652751 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.652769 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.652804 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.652818 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.652935 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.653338 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.653396 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:22:01.654218 5537 ts_tablet_manager.cc:1918] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:01.654366 5537 log.cc:1196] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/ebc52ef0a3014697bfead407187dbd04
I20260620 23:22:01.654572 5228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:01.654923 5537 ts_tablet_manager.cc:1939] T ebc52ef0a3014697bfead407187dbd04 P 759db7158de74db5b84632e6987375f6: Deleting consensus metadata
I20260620 23:22:01.656685 5031 catalog_manager.cc:4841] TS 759db7158de74db5b84632e6987375f6 (127.0.97.2:38417): tablet ebc52ef0a3014697bfead407187dbd04 (table test-table [id=73f4b01c44bf476d869d0af0166cc3be]) successfully deleted
I20260620 23:22:01.657248 5538 ts_tablet_manager.cc:1918] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:01.657416 5538 log.cc:1196] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/ebc52ef0a3014697bfead407187dbd04
I20260620 23:22:01.657826 5538 ts_tablet_manager.cc:1939] T ebc52ef0a3014697bfead407187dbd04 P d443c357eefe44198d3d0425172aed54: Deleting consensus metadata
I20260620 23:22:01.659215 5031 catalog_manager.cc:4841] TS d443c357eefe44198d3d0425172aed54 (127.0.97.3:33941): tablet ebc52ef0a3014697bfead407187dbd04 (table test-table [id=73f4b01c44bf476d869d0af0166cc3be]) successfully deleted
I20260620 23:22:01.745770 5539 ts_tablet_manager.cc:1939] T ebc52ef0a3014697bfead407187dbd04 P cee28583cddb42faad56ade0064e6573: Deleting consensus metadata
I20260620 23:22:01.745771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 5106
W20260620 23:22:01.767822 5034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35929 (error 108)
I20260620 23:22:01.767975 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 5236
I20260620 23:22:01.769888 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.769299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35929 (local address 127.0.0.1:36970)
0620 23:22:01.769609 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.769737 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
W20260620 23:22:01.770107 5034 catalog_manager.cc:4584] TS cee28583cddb42faad56ade0064e6573 (127.0.97.1:35929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ebc52ef0a3014697bfead407187dbd04: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35929: connect: Connection refused (error 111)
I20260620 23:22:01.770924 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.768934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.0.97.2:38417)
0620 23:22:01.769111 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.769117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.769135 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:22:01.769196 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.769202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.769257 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.769376 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.770289 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.770294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.770299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.770316 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.770352 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.770367 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.770482 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.770560 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.770608 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:22:01.771579 5358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:01.823664 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.823134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35929 (local address 127.0.0.1:36982)
0620 23:22:01.823351 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.823464 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 23:22:01.824667 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.824299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35929 (local address 127.0.0.1:36998)
0620 23:22:01.824442 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.824529 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:22:01.876247 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 5236
I20260620 23:22:01.885093 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.884651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35929 (local address 127.0.0.1:37000)
0620 23:22:01.884850 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.884947 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 23:22:01.886025 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.885711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35929 (local address 127.0.0.1:37016)
0620 23:22:01.885850 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.885919 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 23:22:01.894496 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 5366
I20260620 23:22:01.897014 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.895244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.0.97.3:33941)
0620 23:22:01.895454 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:01.895459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:01.895483 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:22:01.895552 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:01.895556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:01.895645 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:01.895739 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:01.896475 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:01.896480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:01.896483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:01.896498 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:01.896527 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:01.896538 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:01.896645 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:01.896713 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 23:22:01.896753 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:22:01.897593 5488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:01.966553 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.966119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35929 (local address 127.0.0.1:37028)
0620 23:22:01.966341 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.966423 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 23:22:01.967490 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:01.967156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35929 (local address 127.0.0.1:37036)
0620 23:22:01.967339 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:01.967393 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:22:01.984181 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 5366
I20260620 23:22:02.001796 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 5016
I20260620 23:22:02.003019 5077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:02.104266 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 5016
2026-06-20T23:22:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2474 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 23:22:02.143581 388 mini_oidc.cc:152] Starting JWKS server
I20260620 23:22:02.145995 388 webserver.cc:460] Webserver started at https://127.0.0.1:42681/ using document root <none> and password file <none>
I20260620 23:22:02.146306 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:22:02.146943 388 webserver.cc:460] Webserver started at http://127.0.0.1:42879/ using document root <none> and password file <none>
2026-06-20T23:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:02Z Disabled control of system clock
I20260620 23:22:02.171180 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44029
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44029
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42681/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/testchainca.cert with env {}
W20260620 23:22:02.479446 5551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:02.479677 5551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:02.479763 5551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:02.486442 5551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:02.486528 5551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:02.486564 5551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:02.486611 5551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:02.486653 5551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:22:02.486763 5551 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:22:02.486806 5551 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42681/jwks.json
I20260620 23:22:02.495990 5551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42681/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:02.497104 5551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:02.498211 5551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:02.508507 5556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:02.508545 5557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:02.508574 5559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:02.509486 5551 server_base.cc:1034] running on GCE node
I20260620 23:22:02.510216 5551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:02.511446 5551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:02.512624 5551 hybrid_clock.cc:648] HybridClock initialized: now 1781997722512567 us; error 51 us; skew 500 ppm
I20260620 23:22:02.512933 5551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:02.515020 5551 webserver.cc:460] Webserver started at http://127.0.97.62:39763/ using document root <none> and password file <none>
I20260620 23:22:02.515647 5551 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:02.515764 5551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:02.516057 5551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:02.518613 5551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "45702b97ec6b403693cd4b2ac90fead3"
format_stamp: "Formatted at 2026-06-20 23:22:02 on dist-test-slave-5kzz"
server_key: "545b9b7a49d3f26705639558524c2cbe"
server_key_iv: "ce4c2cdad56918b4c6d49bad3f8dbd45"
server_key_version: "encryptionkey@0"
I20260620 23:22:02.519428 5551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "45702b97ec6b403693cd4b2ac90fead3"
format_stamp: "Formatted at 2026-06-20 23:22:02 on dist-test-slave-5kzz"
server_key: "545b9b7a49d3f26705639558524c2cbe"
server_key_iv: "ce4c2cdad56918b4c6d49bad3f8dbd45"
server_key_version: "encryptionkey@0"
I20260620 23:22:02.523913 5551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:22:02.527072 5565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:02.528213 5551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:02.528395 5551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "45702b97ec6b403693cd4b2ac90fead3"
format_stamp: "Formatted at 2026-06-20 23:22:02 on dist-test-slave-5kzz"
server_key: "545b9b7a49d3f26705639558524c2cbe"
server_key_iv: "ce4c2cdad56918b4c6d49bad3f8dbd45"
server_key_version: "encryptionkey@0"
I20260620 23:22:02.528594 5551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:02.544986 5551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:02.545737 5551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:02.545993 5551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:02.563263 5551 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44029
I20260620 23:22:02.563278 5616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44029 every 8 connection(s)
I20260620 23:22:02.564478 5551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:02.565840 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 5551
I20260620 23:22:02.566051 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:22:02.566859 388 external_mini_cluster.cc:1428] Setting key 7e71b15063f9d84d2f49bf7278660694
I20260620 23:22:02.568760 5617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:02.579366 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:02.568650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.0.97.62:44029)
0620 23:22:02.569520 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:02.569530 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:02.569567 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:22:02.571297 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:02.571316 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:02.571724 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:02.572105 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:02.572611 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:02.573843 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:02.575162 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:02.575798 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:02.575986 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:02.577126 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:02.577136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:02.577147 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:02.577819 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:02.577887 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:02.577909 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:02.578030 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:02.578240 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 23:22:02.578491 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":300,"threads_started":1}
I20260620 23:22:02.579571 5617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3: Bootstrap starting.
I20260620 23:22:02.582581 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:22:02.582602 5617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:02.583530 5617 log.cc:824] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:02.586202 5617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3: No bootstrap required, opened a new log
I20260620 23:22:02.593097 5617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45702b97ec6b403693cd4b2ac90fead3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44029 } }
I20260620 23:22:02.593464 5617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:02.593577 5617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45702b97ec6b403693cd4b2ac90fead3, State: Initialized, Role: FOLLOWER
I20260620 23:22:02.594089 5617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45702b97ec6b403693cd4b2ac90fead3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44029 } }
I20260620 23:22:02.594300 5617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:02.594417 5617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:02.594574 5617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:02.596566 5617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45702b97ec6b403693cd4b2ac90fead3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44029 } }
I20260620 23:22:02.597031 5617 leader_election.cc:302] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45702b97ec6b403693cd4b2ac90fead3; no voters:
I20260620 23:22:02.597530 5617 leader_election.cc:288] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:22:02.597653 5622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:02.598323 5622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [term 1 LEADER]: Becoming Leader. State: Replica: 45702b97ec6b403693cd4b2ac90fead3, State: Running, Role: LEADER
I20260620 23:22:02.598788 5622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45702b97ec6b403693cd4b2ac90fead3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44029 } }
I20260620 23:22:02.599457 5617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:02.601601 5624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45702b97ec6b403693cd4b2ac90fead3. Latest consensus state: current_term: 1 leader_uuid: "45702b97ec6b403693cd4b2ac90fead3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45702b97ec6b403693cd4b2ac90fead3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44029 } } }
I20260620 23:22:02.601939 5624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:02.602267 5623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45702b97ec6b403693cd4b2ac90fead3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45702b97ec6b403693cd4b2ac90fead3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44029 } } }
I20260620 23:22:02.602623 5623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:02.604171 5630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:02.608357 5630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:02.615690 5630 catalog_manager.cc:1347] Generated new cluster ID: d06956c935f041f68df7423f1753227f
I20260620 23:22:02.615789 5630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:02.631487 5630 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:22:02.632979 5630 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:02.641044 5630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45702b97ec6b403693cd4b2ac90fead3: Generated new TSK 0
I20260620 23:22:02.641785 5630 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:22:02.678578 5647 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 23:22:02.678764 5647 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:22:02.679569 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:02.659375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.0.97.62:44029)
0620 23:22:02.659573 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:02.659581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:02.660151 (+ 570us) server_negotiation.cc:407] Connection header received
0620 23:22:02.660458 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:02.660462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:02.660524 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:02.660633 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:22:02.661191 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:02.661861 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:02.663941 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:02.664348 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:02.664428 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:02.664973 (+ 545us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:22:02.679130 (+ 14157us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42681/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:22:02.679358 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53324: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42681/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 23:22:02.679694 5620 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53324: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42681/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 23:22:02.681288 5646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:02.659389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44029 (local address 127.0.0.1:53324)
0620 23:22:02.660043 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:02.660068 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:02.660252 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:02.660689 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:02.660694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:02.660706 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:22:02.661049 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:02.661060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:02.662024 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:02.662032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:02.663817 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:02.663832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:02.664459 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:02.664500 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:02.664674 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:02.664742 (+ 68us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:22:02.679371 (+ 14629us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42681/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:22:02.679662 (+ 291us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44029: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42681/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":314,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:44029: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42681/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 23:22:02.682942 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 5551
I20260620 23:22:02.683882 5612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:02.773262 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 5551
2026-06-20T23:22:02Z chronyd exiting
I20260620 23:22:02.822872 388 test_util.cc:175] -----------------------------------------------
I20260620 23:22:02.823033 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997680534866-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (683 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 23:22:02.826539 388 mini_oidc.cc:152] Starting JWKS server
I20260620 23:22:02.828426 388 webserver.cc:460] Webserver started at https://127.0.0.1:45245/ using document root <none> and password file <none>
I20260620 23:22:02.828665 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:22:02.829285 388 webserver.cc:460] Webserver started at http://127.0.0.1:41091/ using document root <none> and password file <none>
2026-06-20T23:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:02Z Disabled control of system clock
I20260620 23:22:02.852939 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45303
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45303
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45245/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/testchainca.cert with env {}
W20260620 23:22:03.158919 5654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:03.159219 5654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:03.159305 5654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:03.166424 5654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:03.166534 5654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:03.166574 5654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:03.166646 5654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:03.166710 5654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:22:03.166766 5654 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:22:03.166806 5654 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45245/jwks.json
I20260620 23:22:03.176422 5654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45245/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:03.177649 5654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:03.178936 5654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:03.190840 5659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:03.190924 5660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:03.190881 5662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:03.191658 5654 server_base.cc:1034] running on GCE node
I20260620 23:22:03.192370 5654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:03.193722 5654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:03.194932 5654 hybrid_clock.cc:648] HybridClock initialized: now 1781997723194897 us; error 50 us; skew 500 ppm
I20260620 23:22:03.195276 5654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:03.197475 5654 webserver.cc:460] Webserver started at http://127.0.97.62:46037/ using document root <none> and password file <none>
I20260620 23:22:03.198137 5654 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:03.198278 5654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:03.198589 5654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:03.201386 5654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "ded39d6f80cb433aa37e338772f78bc5"
format_stamp: "Formatted at 2026-06-20 23:22:03 on dist-test-slave-5kzz"
server_key: "b98531a9d4a337bb9868c429856e4de6"
server_key_iv: "8ddf754eeb5dd11dd8a882e4acbe226c"
server_key_version: "encryptionkey@0"
I20260620 23:22:03.202163 5654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "ded39d6f80cb433aa37e338772f78bc5"
format_stamp: "Formatted at 2026-06-20 23:22:03 on dist-test-slave-5kzz"
server_key: "b98531a9d4a337bb9868c429856e4de6"
server_key_iv: "8ddf754eeb5dd11dd8a882e4acbe226c"
server_key_version: "encryptionkey@0"
I20260620 23:22:03.207041 5654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:22:03.210162 5668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:03.211426 5654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:03.211609 5654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "ded39d6f80cb433aa37e338772f78bc5"
format_stamp: "Formatted at 2026-06-20 23:22:03 on dist-test-slave-5kzz"
server_key: "b98531a9d4a337bb9868c429856e4de6"
server_key_iv: "8ddf754eeb5dd11dd8a882e4acbe226c"
server_key_version: "encryptionkey@0"
I20260620 23:22:03.211812 5654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:03.229115 5654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:03.229959 5654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:03.230237 5654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:03.247766 5654 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45303
I20260620 23:22:03.247785 5719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45303 every 8 connection(s)
I20260620 23:22:03.249105 5654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:03.252681 5720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:03.259050 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 5654
I20260620 23:22:03.259250 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:22:03.259860 388 external_mini_cluster.cc:1428] Setting key 93af1b83fe891d91b242ee03af4467cc
I20260620 23:22:03.262600 5720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5: Bootstrap starting.
I20260620 23:22:03.265579 5720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:03.266621 5720 log.cc:824] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:03.269447 5720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5: No bootstrap required, opened a new log
I20260620 23:22:03.272776 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:03.261612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.0.97.62:45303)
0620 23:22:03.262505 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:03.262522 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:03.262560 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:22:03.264268 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:03.264288 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:03.264709 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:03.265090 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:03.265609 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:03.267015 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:03.268665 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:03.269261 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:03.269409 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:03.270561 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:03.270568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:03.270592 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:03.271171 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:03.271251 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:03.271271 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:03.271359 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:03.271581 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 23:22:03.271900 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":271,"threads_started":1}
I20260620 23:22:03.276337 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:22:03.277506 5720 raft_consensus.cc:348] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded39d6f80cb433aa37e338772f78bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45303 } }
I20260620 23:22:03.277824 5720 raft_consensus.cc:374] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:03.277915 5720 raft_consensus.cc:729] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded39d6f80cb433aa37e338772f78bc5, State: Initialized, Role: FOLLOWER
I20260620 23:22:03.278476 5720 consensus_queue.cc:260] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded39d6f80cb433aa37e338772f78bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45303 } }
I20260620 23:22:03.278769 5720 raft_consensus.cc:388] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:03.278936 5720 raft_consensus.cc:482] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:03.279137 5720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:03.281323 5720 raft_consensus.cc:504] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded39d6f80cb433aa37e338772f78bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45303 } }
I20260620 23:22:03.281805 5720 leader_election.cc:302] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ded39d6f80cb433aa37e338772f78bc5; no voters:
I20260620 23:22:03.282332 5720 leader_election.cc:288] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:22:03.282462 5725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:03.283142 5725 raft_consensus.cc:686] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [term 1 LEADER]: Becoming Leader. State: Replica: ded39d6f80cb433aa37e338772f78bc5, State: Running, Role: LEADER
I20260620 23:22:03.283598 5725 consensus_queue.cc:237] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded39d6f80cb433aa37e338772f78bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45303 } }
I20260620 23:22:03.284277 5720 sys_catalog.cc:564] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:03.286034 5726 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ded39d6f80cb433aa37e338772f78bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded39d6f80cb433aa37e338772f78bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45303 } } }
I20260620 23:22:03.286151 5727 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ded39d6f80cb433aa37e338772f78bc5. Latest consensus state: current_term: 1 leader_uuid: "ded39d6f80cb433aa37e338772f78bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded39d6f80cb433aa37e338772f78bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45303 } } }
I20260620 23:22:03.286444 5726 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:03.286444 5727 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:03.287384 5730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:03.292402 5730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:03.300993 5730 catalog_manager.cc:1347] Generated new cluster ID: 3f04a3eedd8941b4b26d4edb2fb6d8e2
I20260620 23:22:03.301093 5730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:03.319054 5730 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:22:03.320812 5730 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:03.328030 5730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ded39d6f80cb433aa37e338772f78bc5: Generated new TSK 0
I20260620 23:22:03.328905 5730 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:22:03.361605 5750 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 23:22:03.361752 5750 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:22:03.363022 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:03.342654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40496 (local address 127.0.97.62:45303)
0620 23:22:03.342856 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:03.342863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:03.343391 (+ 528us) server_negotiation.cc:407] Connection header received
0620 23:22:03.343595 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:03.343603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:03.343699 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:03.343872 (+ 173us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:22:03.344330 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:03.345436 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:03.346870 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:03.347584 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:03.347737 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:03.348225 (+ 488us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:22:03.362288 (+ 14063us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45245/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:22:03.362663 (+ 375us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40496: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45245/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 23:22:03.363152 5723 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40496: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45245/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 23:22:03.363142 5749 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:03.342613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45303 (local address 127.0.0.1:40496)
0620 23:22:03.343289 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:03.343306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:03.343422 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:03.343884 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:03.343887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:03.343896 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:22:03.344158 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:03.344165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:03.345573 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:03.345577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:03.346651 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:03.346663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:03.347730 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:03.347787 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:03.347920 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:03.347952 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:22:03.362552 (+ 14600us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45245/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:22:03.362818 (+ 266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45245/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":397,"threads_started":1}
I20260620 23:22:03.364593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 5654
I20260620 23:22:03.365859 5715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:03.453367 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 5654
2026-06-20T23:22:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (680 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 23:22:03.507014 388 mini_oidc.cc:152] Starting JWKS server
I20260620 23:22:03.508847 388 webserver.cc:460] Webserver started at https://127.0.0.1:41471/ using document root <none> and password file <none>
I20260620 23:22:03.509097 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:22:03.509725 388 webserver.cc:460] Webserver started at http://127.0.0.1:45351/ using document root <none> and password file <none>
2026-06-20T23:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:03Z Disabled control of system clock
I20260620 23:22:03.534101 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44179
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44179
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41471/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:22:03.837558 5757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:03.837821 5757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:03.837903 5757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:03.844921 5757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:03.845021 5757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:03.845067 5757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:03.845117 5757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:03.845166 5757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:22:03.845227 5757 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:22:03.845268 5757 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41471/jwks.json
I20260620 23:22:03.853902 5757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41471/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:03.855046 5757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:03.856684 5757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:03.867771 5762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:03.867857 5763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:03.867825 5765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:03.868789 5757 server_base.cc:1034] running on GCE node
I20260620 23:22:03.869549 5757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:03.870880 5757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:03.872094 5757 hybrid_clock.cc:648] HybridClock initialized: now 1781997723872074 us; error 41 us; skew 500 ppm
I20260620 23:22:03.872422 5757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:03.877776 5757 webserver.cc:460] Webserver started at http://127.0.97.62:40549/ using document root <none> and password file <none>
I20260620 23:22:03.878461 5757 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:03.878593 5757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:03.878948 5757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:03.881872 5757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "41367c86b664465eaa775182b2efc156"
format_stamp: "Formatted at 2026-06-20 23:22:03 on dist-test-slave-5kzz"
server_key: "24d44b004d60d8a462c7f791796a36a6"
server_key_iv: "b1e068f43c4969c5779981dc70d27a5b"
server_key_version: "encryptionkey@0"
I20260620 23:22:03.882632 5757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "41367c86b664465eaa775182b2efc156"
format_stamp: "Formatted at 2026-06-20 23:22:03 on dist-test-slave-5kzz"
server_key: "24d44b004d60d8a462c7f791796a36a6"
server_key_iv: "b1e068f43c4969c5779981dc70d27a5b"
server_key_version: "encryptionkey@0"
I20260620 23:22:03.887471 5757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:22:03.890615 5771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:03.891857 5757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 23:22:03.892021 5757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "41367c86b664465eaa775182b2efc156"
format_stamp: "Formatted at 2026-06-20 23:22:03 on dist-test-slave-5kzz"
server_key: "24d44b004d60d8a462c7f791796a36a6"
server_key_iv: "b1e068f43c4969c5779981dc70d27a5b"
server_key_version: "encryptionkey@0"
I20260620 23:22:03.892216 5757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:03.919390 5757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:03.920224 5757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:03.920502 5757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:03.937702 5757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44179
I20260620 23:22:03.937721 5822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44179 every 8 connection(s)
I20260620 23:22:03.939051 5757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:03.940461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 5757
I20260620 23:22:03.940659 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:22:03.941409 388 external_mini_cluster.cc:1428] Setting key 0efe612a674af28e48edddbb53401c8c
I20260620 23:22:03.943607 5823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:03.955245 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:03.943281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.0.97.62:44179)
0620 23:22:03.944222 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:03.944233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:03.944272 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:22:03.945826 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:03.945847 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:03.946285 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:03.946670 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:03.947238 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:03.948809 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:03.950575 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:03.951605 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:03.951733 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:03.952921 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:03.952928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:03.952940 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:03.953564 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:03.953638 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:03.953660 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:03.953748 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:03.953976 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 23:22:03.954300 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":345,"threads_started":1}
I20260620 23:22:03.955905 5823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156: Bootstrap starting.
I20260620 23:22:03.958777 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:22:03.959017 5823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:03.960136 5823 log.cc:824] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:03.962747 5823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156: No bootstrap required, opened a new log
I20260620 23:22:03.969363 5823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41367c86b664465eaa775182b2efc156" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44179 } }
I20260620 23:22:03.969666 5823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:03.969738 5823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41367c86b664465eaa775182b2efc156, State: Initialized, Role: FOLLOWER
I20260620 23:22:03.970247 5823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41367c86b664465eaa775182b2efc156" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44179 } }
I20260620 23:22:03.970423 5823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:03.970515 5823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:03.970652 5823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:03.972854 5823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41367c86b664465eaa775182b2efc156" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44179 } }
I20260620 23:22:03.973260 5823 leader_election.cc:302] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41367c86b664465eaa775182b2efc156; no voters:
I20260620 23:22:03.973766 5823 leader_election.cc:288] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:22:03.973889 5828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:03.974614 5828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [term 1 LEADER]: Becoming Leader. State: Replica: 41367c86b664465eaa775182b2efc156, State: Running, Role: LEADER
I20260620 23:22:03.975136 5828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41367c86b664465eaa775182b2efc156" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44179 } }
I20260620 23:22:03.975819 5823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:03.977828 5829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41367c86b664465eaa775182b2efc156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41367c86b664465eaa775182b2efc156" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44179 } } }
I20260620 23:22:03.978163 5829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:03.979056 5830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41367c86b664465eaa775182b2efc156. Latest consensus state: current_term: 1 leader_uuid: "41367c86b664465eaa775182b2efc156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41367c86b664465eaa775182b2efc156" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44179 } } }
I20260620 23:22:03.979452 5830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:03.981092 5834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:03.985574 5834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:03.993217 5834 catalog_manager.cc:1347] Generated new cluster ID: 2bd3130450024b2f94aa5c2026ddb46f
I20260620 23:22:03.993352 5834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:04.004747 5834 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:22:04.006403 5834 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:04.014074 5834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41367c86b664465eaa775182b2efc156: Generated new TSK 0
I20260620 23:22:04.015039 5834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:22:04.040849 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:04.032932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.0.97.62:44179)
0620 23:22:04.033150 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:04.033156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:04.033526 (+ 370us) server_negotiation.cc:407] Connection header received
0620 23:22:04.033771 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:04.033775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:04.033846 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:04.033959 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:22:04.034474 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:04.035216 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:04.037186 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:04.037574 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:04.037655 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:04.038172 (+ 517us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:22:04.040313 (+ 2141us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:22:04.040575 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33362: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 23:22:04.040953 5826 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33362: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 23:22:04.042802 5852 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:04.032825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44179 (local address 127.0.0.1:33362)
0620 23:22:04.033421 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:04.033441 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:04.033588 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:04.034006 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:04.034013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:04.034027 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:22:04.034337 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:04.034349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:04.035351 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:04.035356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:04.037069 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:04.037083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:04.037678 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:04.037748 (+ 70us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:04.037931 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:04.037971 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:22:04.040501 (+ 2530us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:22:04.042327 (+ 1826us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44179: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":1577,"thread_start_us":307,"threads_started":1}
I20260620 23:22:04.044494 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 5757
I20260620 23:22:04.045521 5818 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 23:22:04.052434 5853 webserver.cc:533] Webserver: sslize failed:
E20260620 23:22:04.052511 5853 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:22:04.132023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 5757
2026-06-20T23:22:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (678 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T23:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:04Z Disabled control of system clock
I20260620 23:22:04.209241 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33031
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33031
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/testchainca.cert with env {}
W20260620 23:22:04.510432 5858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:04.510751 5858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:04.510900 5858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:04.517947 5858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:04.518038 5858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:04.518105 5858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:04.518210 5858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:04.518293 5858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:22:04.518406 5858 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:22:04.518498 5858 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 23:22:04.527967 5858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:04.529241 5858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:04.530550 5858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:04.542013 5863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:04.541973 5866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:04.542047 5864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:04.543012 5858 server_base.cc:1034] running on GCE node
I20260620 23:22:04.543692 5858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:04.544894 5858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:04.546111 5858 hybrid_clock.cc:648] HybridClock initialized: now 1781997724546072 us; error 53 us; skew 500 ppm
I20260620 23:22:04.546442 5858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:04.548506 5858 webserver.cc:460] Webserver started at http://127.0.97.62:43917/ using document root <none> and password file <none>
I20260620 23:22:04.549134 5858 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:04.549263 5858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:04.549554 5858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:04.552273 5858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "0996e182b5984e02889fabf80caf00a3"
format_stamp: "Formatted at 2026-06-20 23:22:04 on dist-test-slave-5kzz"
server_key: "dd34afca1864127f6ad9557f534f0447"
server_key_iv: "5eb93b6ca429c7608a33583a41984829"
server_key_version: "encryptionkey@0"
I20260620 23:22:04.553045 5858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "0996e182b5984e02889fabf80caf00a3"
format_stamp: "Formatted at 2026-06-20 23:22:04 on dist-test-slave-5kzz"
server_key: "dd34afca1864127f6ad9557f534f0447"
server_key_iv: "5eb93b6ca429c7608a33583a41984829"
server_key_version: "encryptionkey@0"
I20260620 23:22:04.557680 5858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260620 23:22:04.560804 5872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:04.562013 5858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:22:04.562209 5858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "0996e182b5984e02889fabf80caf00a3"
format_stamp: "Formatted at 2026-06-20 23:22:04 on dist-test-slave-5kzz"
server_key: "dd34afca1864127f6ad9557f534f0447"
server_key_iv: "5eb93b6ca429c7608a33583a41984829"
server_key_version: "encryptionkey@0"
I20260620 23:22:04.562407 5858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:04.583520 5858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:04.584245 5858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:04.584515 5858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:04.601570 5858 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33031
I20260620 23:22:04.601575 5923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33031 every 8 connection(s)
I20260620 23:22:04.602870 5858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:04.605413 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 5858
I20260620 23:22:04.605604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:22:04.606410 388 external_mini_cluster.cc:1428] Setting key f71e85e0324e385540f37f5579652e6d
I20260620 23:22:04.607218 5924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:04.619807 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:04.608249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.0.97.62:33031)
0620 23:22:04.609160 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:04.609170 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:04.609208 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:22:04.611451 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:04.611470 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:04.611899 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:04.612202 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:04.612765 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:04.614040 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:04.615762 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:04.616385 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:04.616511 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:04.617675 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:04.617685 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:04.617697 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:04.618228 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:04.618300 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:04.618322 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:04.618400 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:04.618610 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 23:22:04.618936 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":344,"threads_started":1}
I20260620 23:22:04.620059 5924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3: Bootstrap starting.
I20260620 23:22:04.623489 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:22:04.623499 5924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:04.624439 5924 log.cc:824] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:04.627184 5924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3: No bootstrap required, opened a new log
I20260620 23:22:04.633920 5924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0996e182b5984e02889fabf80caf00a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33031 } }
I20260620 23:22:04.634281 5924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:04.634394 5924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0996e182b5984e02889fabf80caf00a3, State: Initialized, Role: FOLLOWER
I20260620 23:22:04.634974 5924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0996e182b5984e02889fabf80caf00a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33031 } }
I20260620 23:22:04.635201 5924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:04.635349 5924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:04.635522 5924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:04.637501 5924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0996e182b5984e02889fabf80caf00a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33031 } }
I20260620 23:22:04.637946 5924 leader_election.cc:302] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0996e182b5984e02889fabf80caf00a3; no voters:
I20260620 23:22:04.638491 5924 leader_election.cc:288] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:22:04.638633 5929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:04.639328 5929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0996e182b5984e02889fabf80caf00a3, State: Running, Role: LEADER
I20260620 23:22:04.639853 5929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0996e182b5984e02889fabf80caf00a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33031 } }
I20260620 23:22:04.640483 5924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:04.642576 5931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0996e182b5984e02889fabf80caf00a3. Latest consensus state: current_term: 1 leader_uuid: "0996e182b5984e02889fabf80caf00a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0996e182b5984e02889fabf80caf00a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33031 } } }
I20260620 23:22:04.643041 5931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:04.642849 5930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0996e182b5984e02889fabf80caf00a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0996e182b5984e02889fabf80caf00a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33031 } } }
I20260620 23:22:04.643262 5930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:04.651623 5935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:04.656646 5935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:04.664991 5935 catalog_manager.cc:1347] Generated new cluster ID: 39849cc1eb1d4a4d9a14bd6a3b9c47b9
I20260620 23:22:04.665091 5935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:04.672825 5935 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:22:04.673949 5935 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:04.681772 5935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0996e182b5984e02889fabf80caf00a3: Generated new TSK 0
I20260620 23:22:04.682577 5935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:22:05.134598 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:04.703286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.0.97.62:33031)
0620 23:22:04.703486 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:04.703492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:04.703780 (+ 288us) server_negotiation.cc:407] Connection header received
0620 23:22:04.703937 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:04.703944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:04.704030 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:04.704190 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:22:04.704577 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:04.705643 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:04.706992 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:04.707663 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:04.707785 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:04.708217 (+ 432us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:22:05.134071 (+425854us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:22:05.134387 (+ 316us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41942: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 23:22:05.134783 5927 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41942: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 23:22:05.134979 5953 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:04.703120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33031 (local address 127.0.0.1:41942)
0620 23:22:04.703672 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:04.703687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:04.703794 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:04.704204 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:04.704207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:04.704216 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:22:04.704446 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:04.704453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:04.705767 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:04.705770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:04.706868 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:04.706878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:04.707780 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:04.707820 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:04.707949 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:04.707977 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:22:05.134335 (+426358us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:22:05.134567 (+ 232us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33031: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":297,"threads_started":1}
I20260620 23:22:05.136096 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 5858
I20260620 23:22:05.137125 5919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:05.233315 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 5858
2026-06-20T23:22:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1084 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (386 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (387 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:22:06 dist-test-slave-5kzz krb5kdc[5968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:22:06 dist-test-slave-5kzz krb5kdc[5968](info): set up 2 sockets
Jun 20 23:22:06 dist-test-slave-5kzz krb5kdc[5968](info): commencing operation
krb5kdc: starting...
W20260620 23:22:07.071278 388 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:22:07 dist-test-slave-5kzz krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:22:07.251448 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39847
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:07.553931 5981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:07.554189 5981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:07.554284 5981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:07.561124 5981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:07.561226 5981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:07.561278 5981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:07.561326 5981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:07.561369 5981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:22:07.571242 5981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:07.572950 5981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:07.574793 5981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:07.587126 5986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:07.587209 5987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:07.588186 5989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:07.588204 5981 server_base.cc:1034] running on GCE node
I20260620 23:22:07.589027 5981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:07.590394 5981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:07.591578 5981 hybrid_clock.cc:648] HybridClock initialized: now 1781997727591534 us; error 39 us; skew 500 ppm
I20260620 23:22:07.591917 5981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:07 dist-test-slave-5kzz krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:07.594977 5981 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:22:07.596324 5981 webserver.cc:460] Webserver started at http://127.0.97.62:34565/ using document root <none> and password file <none>
I20260620 23:22:07.596968 5981 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:07.597105 5981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:07.597419 5981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:07.600287 5981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "0ec34e662c9442398a6b6e772c163099"
format_stamp: "Formatted at 2026-06-20 23:22:07 on dist-test-slave-5kzz"
server_key: "99656900ec3517141e9e4584c855f86e"
server_key_iv: "6ca1651d78a05dbe41639aab0cffbd82"
server_key_version: "encryptionkey@0"
I20260620 23:22:07.601032 5981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "0ec34e662c9442398a6b6e772c163099"
format_stamp: "Formatted at 2026-06-20 23:22:07 on dist-test-slave-5kzz"
server_key: "99656900ec3517141e9e4584c855f86e"
server_key_iv: "6ca1651d78a05dbe41639aab0cffbd82"
server_key_version: "encryptionkey@0"
I20260620 23:22:07.605854 5981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 23:22:07.609157 5996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:07.610406 5981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:22:07.610581 5981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "0ec34e662c9442398a6b6e772c163099"
format_stamp: "Formatted at 2026-06-20 23:22:07 on dist-test-slave-5kzz"
server_key: "99656900ec3517141e9e4584c855f86e"
server_key_iv: "6ca1651d78a05dbe41639aab0cffbd82"
server_key_version: "encryptionkey@0"
I20260620 23:22:07.610808 5981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:07.629755 5981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:07.633873 5981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:07.634142 5981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:07.651531 5981 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39847
I20260620 23:22:07.651561 6047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39847 every 8 connection(s)
I20260620 23:22:07.652791 5981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:07.656414 6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:07.658555 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 5981
I20260620 23:22:07.658768 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:22:07.659520 388 external_mini_cluster.cc:1428] Setting key b34f432ac61f3d3e34b46faee27fd244
I20260620 23:22:07.669199 6048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099: Bootstrap starting.
I20260620 23:22:07.672468 6048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:07.673405 6048 log.cc:824] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:22:07 dist-test-slave-5kzz krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:22:07.676183 6048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099: No bootstrap required, opened a new log
I20260620 23:22:07.683401 6051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:07.661823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.0.97.62:39847)
0620 23:22:07.662724 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:07.662739 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:07.663132 (+ 393us) server_negotiation.cc:407] Connection header received
0620 23:22:07.665191 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:07.665219 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:07.665735 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:07.666158 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:07.667491 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:07.669325 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:07.670500 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:07.671497 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:07.671675 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:07.675406 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:07.675418 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:07.675436 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:07.675471 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:07.679305 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:07.679912 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:07.679920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:07.679928 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:07.680107 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:07.680426 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:07.680436 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:07.680440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:07.681100 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:07.681409 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:07.681708 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 23:22:07.682139 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":390,"threads_started":1}
I20260620 23:22:07.686342 6048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec34e662c9442398a6b6e772c163099" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39847 } }
I20260620 23:22:07.686728 6048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:07.686843 6048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec34e662c9442398a6b6e772c163099, State: Initialized, Role: FOLLOWER
I20260620 23:22:07.687367 6048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec34e662c9442398a6b6e772c163099" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39847 } }
I20260620 23:22:07.687592 6048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:07.687721 6048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:07.687875 6048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:07.690016 6048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec34e662c9442398a6b6e772c163099" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39847 } }
I20260620 23:22:07.690482 6048 leader_election.cc:302] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ec34e662c9442398a6b6e772c163099; no voters:
I20260620 23:22:07.691102 6048 leader_election.cc:288] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:22:07.691231 6053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:07.692938 6053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [term 1 LEADER]: Becoming Leader. State: Replica: 0ec34e662c9442398a6b6e772c163099, State: Running, Role: LEADER
I20260620 23:22:07.693397 6053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec34e662c9442398a6b6e772c163099" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39847 } }
I20260620 23:22:07.693706 6048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:07.696372 6054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ec34e662c9442398a6b6e772c163099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec34e662c9442398a6b6e772c163099" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39847 } } }
I20260620 23:22:07.696346 6055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ec34e662c9442398a6b6e772c163099. Latest consensus state: current_term: 1 leader_uuid: "0ec34e662c9442398a6b6e772c163099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec34e662c9442398a6b6e772c163099" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39847 } } }
I20260620 23:22:07.696812 6054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:07.696849 6055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:07.698297 6059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:07.703433 6059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:07.711799 6059 catalog_manager.cc:1347] Generated new cluster ID: abc47ec32c57436fb4bb73446b3e95f2
I20260620 23:22:07.711897 6059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:07.732020 6059 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:22:07.733827 6059 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:07.747576 6059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ec34e662c9442398a6b6e772c163099: Generated new TSK 0
I20260620 23:22:07.748659 6059 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:22:07.801590 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39847
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:08.113220 6074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:08.113498 6074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:08.113688 6074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:08.120862 6074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:08.120955 6074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:08.121062 6074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:22:08.130035 6074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:08.131225 6074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:08.132387 6074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:08.144834 6079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:08.144847 6080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:08.144793 6082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:08.145673 6074 server_base.cc:1034] running on GCE node
I20260620 23:22:08.146178 6074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:08.147060 6074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:08.148264 6074 hybrid_clock.cc:648] HybridClock initialized: now 1781997728148234 us; error 50 us; skew 500 ppm
I20260620 23:22:08.148566 6074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:08 dist-test-slave-5kzz krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:08.151445 6074 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:22:08.152719 6074 webserver.cc:460] Webserver started at http://127.0.97.1:46163/ using document root <none> and password file <none>
I20260620 23:22:08.153386 6074 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:08.153520 6074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:08.153815 6074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:08.156623 6074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "e741e136069b4550ac34c633f00d3a89"
format_stamp: "Formatted at 2026-06-20 23:22:08 on dist-test-slave-5kzz"
server_key: "490964642d8bb11575323526afe04436"
server_key_iv: "49b5083edf8d1c665d9bb1680b95e83b"
server_key_version: "encryptionkey@0"
I20260620 23:22:08.157388 6074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e741e136069b4550ac34c633f00d3a89"
format_stamp: "Formatted at 2026-06-20 23:22:08 on dist-test-slave-5kzz"
server_key: "490964642d8bb11575323526afe04436"
server_key_iv: "49b5083edf8d1c665d9bb1680b95e83b"
server_key_version: "encryptionkey@0"
I20260620 23:22:08.162031 6074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260620 23:22:08.165213 6089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:08.166354 6074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 23:22:08.166514 6074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "e741e136069b4550ac34c633f00d3a89"
format_stamp: "Formatted at 2026-06-20 23:22:08 on dist-test-slave-5kzz"
server_key: "490964642d8bb11575323526afe04436"
server_key_iv: "49b5083edf8d1c665d9bb1680b95e83b"
server_key_version: "encryptionkey@0"
I20260620 23:22:08.166714 6074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:08.176982 6074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:08.180843 6074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:08.181103 6074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:08.181919 6074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:08.183177 6074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:08.183277 6074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:08.183413 6074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:08.183477 6074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:08.211434 6074 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44017
I20260620 23:22:08.211465 6201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44017 every 8 connection(s)
I20260620 23:22:08.212652 6074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:22:08.218995 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 6074
I20260620 23:22:08.219177 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:22:08.219794 388 external_mini_cluster.cc:1428] Setting key 63234e4e07a19b3f5f181f0c85ca6e1c
Jun 20 23:22:08 dist-test-slave-5kzz krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:22:08.230989 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:08.215128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45485 (local address 127.0.97.62:39847)
0620 23:22:08.215533 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:08.215538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:08.216443 (+ 905us) server_negotiation.cc:407] Connection header received
0620 23:22:08.218121 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:08.218127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:08.218213 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:08.218325 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:08.219830 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:08.220567 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:08.221993 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:08.222375 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:08.222499 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:08.225621 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:08.225625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:08.225629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:08.225656 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:08.228241 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:08.228905 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:08.228912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:08.228916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:08.229024 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:08.229409 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:08.229416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:08.229421 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:08.229660 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:08.229801 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:08.230409 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 23:22:08.230619 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":215,"threads_started":1}
I20260620 23:22:08.231906 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:08.215617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39847 (local address 127.0.97.1:45485)
0620 23:22:08.216294 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:08.216337 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:08.217883 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:08.218544 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:08.218552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:08.219105 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:08.219674 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:08.219689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:08.220706 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:08.220711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:08.221859 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:08.221874 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:08.222573 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:08.222580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:08.222844 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:08.223472 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:08.223492 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:08.225424 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:08.228379 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:08.228387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:08.228402 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:08.228793 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:08.229133 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:08.229138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:08.229141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:08.229311 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:08.229783 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:08.229790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:08.230091 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 23:22:08.230341 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 23:22:08.230834 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":273,"threads_started":1}
I20260620 23:22:08.233387 6202 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39847
I20260620 23:22:08.233665 6202 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:08.234297 6202 heartbeater.cc:511] Master 127.0.97.62:39847 requested a full tablet report, sending...
I20260620 23:22:08.236593 6013 ts_manager.cc:194] Registered new tserver with Master: e741e136069b4550ac34c633f00d3a89 (127.0.97.1:44017)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:22:08.238201 6013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:22:08.261222 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39847
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:08.567940 6208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:08.568195 6208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:08.568282 6208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:08.575367 6208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:08.575484 6208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:08.575590 6208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:22:08.584800 6208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:08.585913 6208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:08.587196 6208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:08.599622 6213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:08.599581 6216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:08.599655 6214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:08.600401 6208 server_base.cc:1034] running on GCE node
I20260620 23:22:08.600903 6208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:08.601715 6208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:08.602890 6208 hybrid_clock.cc:648] HybridClock initialized: now 1781997728602837 us; error 50 us; skew 500 ppm
I20260620 23:22:08.603164 6208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:08 dist-test-slave-5kzz krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:08.606050 6208 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:22:08.607481 6208 webserver.cc:460] Webserver started at http://127.0.97.2:45003/ using document root <none> and password file <none>
I20260620 23:22:08.608090 6208 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:08.608218 6208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:08.608498 6208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:08.611239 6208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "590ee9f6e73844d2837b43a7e15ea425"
format_stamp: "Formatted at 2026-06-20 23:22:08 on dist-test-slave-5kzz"
server_key: "63ae4be10a8cd5f096ab7b92dab662f5"
server_key_iv: "4a6b9ef2fda17d0c980cc4548b9254e1"
server_key_version: "encryptionkey@0"
I20260620 23:22:08.611930 6208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "590ee9f6e73844d2837b43a7e15ea425"
format_stamp: "Formatted at 2026-06-20 23:22:08 on dist-test-slave-5kzz"
server_key: "63ae4be10a8cd5f096ab7b92dab662f5"
server_key_iv: "4a6b9ef2fda17d0c980cc4548b9254e1"
server_key_version: "encryptionkey@0"
I20260620 23:22:08.616694 6208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:22:08.619792 6223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:08.620960 6208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:08.621115 6208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "590ee9f6e73844d2837b43a7e15ea425"
format_stamp: "Formatted at 2026-06-20 23:22:08 on dist-test-slave-5kzz"
server_key: "63ae4be10a8cd5f096ab7b92dab662f5"
server_key_iv: "4a6b9ef2fda17d0c980cc4548b9254e1"
server_key_version: "encryptionkey@0"
I20260620 23:22:08.621320 6208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:08.639712 6208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:08.643144 6208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:08.643440 6208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:08.644313 6208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:08.645530 6208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:08.645623 6208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:08.645774 6208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:08.645838 6208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:08.678161 6208 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36409
I20260620 23:22:08.678164 6335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36409 every 8 connection(s)
I20260620 23:22:08.679390 6208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:22:08.688441 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 6208
I20260620 23:22:08.688606 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:22:08.689024 388 external_mini_cluster.cc:1428] Setting key 498461cb20a6ffdabc8151b8f09c48df
Jun 20 23:22:08 dist-test-slave-5kzz krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:22:08.698911 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:08.681684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59761 (local address 127.0.97.62:39847)
0620 23:22:08.681834 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:08.681839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:08.682979 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 23:22:08.684646 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:08.684651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:08.684745 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:08.684845 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:08.686183 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:08.686955 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:08.688418 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:08.688879 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:08.689002 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:08.692928 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:08.692932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:08.692936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:08.692983 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:08.695625 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:08.696527 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:08.696531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:08.696533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:08.696645 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:08.697058 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:08.697065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:08.697070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:08.697411 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:08.697571 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:08.698198 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 23:22:08.698462 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:22:08.699599 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:08.682154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39847 (local address 127.0.97.2:59761)
0620 23:22:08.682844 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:08.682878 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:08.684407 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:08.685057 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:08.685064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:08.685511 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:08.686035 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:08.686049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:08.687088 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:08.687098 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:08.688283 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:08.688294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:08.689118 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:08.689125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:08.689334 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:08.689977 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:08.690016 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:08.692732 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:08.695750 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:08.695761 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:08.695779 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:08.696423 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:08.696788 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:08.696792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:08.696795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:08.696959 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:08.697539 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:08.697544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:08.697817 (+ 273us) client_negotiation.cc:769] Sending connection context
0620 23:22:08.698038 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 23:22:08.698486 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":270,"threads_started":1}
I20260620 23:22:08.700927 6336 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39847
I20260620 23:22:08.701189 6336 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:08.701861 6336 heartbeater.cc:511] Master 127.0.97.62:39847 requested a full tablet report, sending...
I20260620 23:22:08.703159 6013 ts_manager.cc:194] Registered new tserver with Master: 590ee9f6e73844d2837b43a7e15ea425 (127.0.97.2:36409)
I20260620 23:22:08.703896 6013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59761
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:22:08.730352 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39847
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:09.032305 6341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:09.032598 6341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:09.032759 6341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:09.039767 6341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:09.039913 6341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:09.040094 6341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:22:09.049625 6341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:09.050854 6341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:09.052102 6341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:09.064702 6346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:09.064739 6347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:09.064672 6349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:09.065696 6341 server_base.cc:1034] running on GCE node
I20260620 23:22:09.066193 6341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:09.067067 6341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:09.068290 6341 hybrid_clock.cc:648] HybridClock initialized: now 1781997729068258 us; error 52 us; skew 500 ppm
I20260620 23:22:09.068583 6341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:09.076074 6341 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:22:09.077369 6341 webserver.cc:460] Webserver started at http://127.0.97.3:34239/ using document root <none> and password file <none>
I20260620 23:22:09.077999 6341 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:09.078128 6341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:09.078405 6341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:09.081024 6341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "07837be2c91248cbb9f9a430faf2d4e8"
format_stamp: "Formatted at 2026-06-20 23:22:09 on dist-test-slave-5kzz"
server_key: "3cd9832a66e08923740279e5eda87174"
server_key_iv: "cd640df4ffcba85c174b8f4f6949584d"
server_key_version: "encryptionkey@0"
I20260620 23:22:09.081759 6341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "07837be2c91248cbb9f9a430faf2d4e8"
format_stamp: "Formatted at 2026-06-20 23:22:09 on dist-test-slave-5kzz"
server_key: "3cd9832a66e08923740279e5eda87174"
server_key_iv: "cd640df4ffcba85c174b8f4f6949584d"
server_key_version: "encryptionkey@0"
I20260620 23:22:09.086477 6341 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 23:22:09.089555 6356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:09.090763 6341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:09.090915 6341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "07837be2c91248cbb9f9a430faf2d4e8"
format_stamp: "Formatted at 2026-06-20 23:22:09 on dist-test-slave-5kzz"
server_key: "3cd9832a66e08923740279e5eda87174"
server_key_iv: "cd640df4ffcba85c174b8f4f6949584d"
server_key_version: "encryptionkey@0"
I20260620 23:22:09.091090 6341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:09.109520 6341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:09.113027 6341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:09.113296 6341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:09.114149 6341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:09.115458 6341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:09.115543 6341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:09.115676 6341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:09.115741 6341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:09.145871 6341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37511
I20260620 23:22:09.145907 6468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37511 every 8 connection(s)
I20260620 23:22:09.147135 6341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:22:09.157379 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 6341
I20260620 23:22:09.157536 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:22:09.157961 388 external_mini_cluster.cc:1428] Setting key 16f3a9004ccaa3095e2853cfc7825b5e
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:22:09.165550 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:09.149452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56579 (local address 127.0.97.62:39847)
0620 23:22:09.149614 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:09.149619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:09.150881 (+ 1262us) server_negotiation.cc:407] Connection header received
0620 23:22:09.152512 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:09.152517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:09.152593 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:09.152689 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:09.154041 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.154840 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.156325 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.156711 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.156835 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:09.160267 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:09.160272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:09.160275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:09.160306 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:09.162931 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.163569 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.163577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.163582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.163688 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.164065 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.164072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.164075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.164318 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:09.164454 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:09.165045 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 23:22:09.165248 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:22:09.166491 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:09.149920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39847 (local address 127.0.97.3:56579)
0620 23:22:09.150671 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:09.150780 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:09.152276 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:09.152898 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:09.152905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:09.153329 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:09.153869 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:09.153883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.154984 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.154988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:09.156172 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:09.156183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.156922 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.156930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:09.157147 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:09.157797 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:09.157818 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:09.160074 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:09.163053 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:09.163061 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:09.163076 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:09.163458 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:09.163786 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:09.163792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:09.163797 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:09.163967 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:09.164429 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:09.164436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:09.164725 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 23:22:09.164970 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 23:22:09.165432 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":300,"threads_started":1}
I20260620 23:22:09.167727 6469 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39847
I20260620 23:22:09.167994 6469 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:09.168618 6469 heartbeater.cc:511] Master 127.0.97.62:39847 requested a full tablet report, sending...
I20260620 23:22:09.169888 6013 ts_manager.cc:194] Registered new tserver with Master: 07837be2c91248cbb9f9a430faf2d4e8 (127.0.97.3:37511)
I20260620 23:22:09.170621 6013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56579
I20260620 23:22:09.172546 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/test-admin.keytab.
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:09.195858 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 23:22:09.200376 6476 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:09.198140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.0.97.1:44017)
0620 23:22:09.198665 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:09.198709 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:09.198743 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:22:09.198863 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:09.198870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:09.199105 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:09.199225 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:09.200021 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52480: BlockingRecv error: recv got EOF from 127.0.0.1:52480 (error 108)
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":297,"threads_started":1}
W20260620 23:22:09.200506 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:09.201893 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:09.203183 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:22:09.218240 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:09.204609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.0.97.1:44017)
0620 23:22:09.204782 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:09.204788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:09.204952 (+ 164us) server_negotiation.cc:407] Connection header received
0620 23:22:09.205140 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:09.205144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:09.205195 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:09.205292 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:09.206171 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.207206 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.208400 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.208898 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.209037 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:09.212466 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:09.212476 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:09.212492 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:09.212530 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:09.215767 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.216368 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.216381 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.216383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.216498 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.216851 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.216857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.216860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.217164 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:09.217464 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:09.217784 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 23:22:09.218007 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 23:22:09.223392 6476 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:09.221671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.0.97.1:44017)
0620 23:22:09.221847 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:09.221854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:09.222152 (+ 298us) server_negotiation.cc:407] Connection header received
0620 23:22:09.222330 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:09.222333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:09.222391 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:09.222481 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:09.223177 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52490: BlockingRecv error: recv got EOF from 127.0.0.1:52490 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 23:22:09.223577 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:09.224972 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:09.226205 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:22:09.238523 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:09.227379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.0.97.1:44017)
0620 23:22:09.227563 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:09.227568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:09.227816 (+ 248us) server_negotiation.cc:407] Connection header received
0620 23:22:09.227928 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:09.227931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:09.227977 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:09.228048 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:09.228882 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.229513 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.230574 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.231008 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.231095 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:09.234196 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:09.234200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:09.234204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:09.234229 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:09.236622 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.237112 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.237115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.237117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.237213 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.237515 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.237518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.237520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.237737 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:09.237899 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:09.238092 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 23:22:09.238254 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:22:09.241545 6202 heartbeater.cc:503] Master 127.0.97.62:39847 was elected leader, sending a full tablet report...
W20260620 23:22:09.241978 6476 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:09.240702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.0.97.1:44017)
0620 23:22:09.240869 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:09.240874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:09.240977 (+ 103us) server_negotiation.cc:407] Connection header received
0620 23:22:09.241085 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:09.241088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:09.241145 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:09.241213 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:09.241756 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52506: BlockingRecv error: recv got EOF from 127.0.0.1:52506 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 23:22:09.242085 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997680534866-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:09.243583 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:09.244766 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:22:09.256889 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:09.245872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.0.97.1:44017)
0620 23:22:09.246041 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:09.246046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:09.246182 (+ 136us) server_negotiation.cc:407] Connection header received
0620 23:22:09.246287 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:09.246290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:09.246354 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:09.246425 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:09.247288 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.247942 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.249621 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.249977 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.250061 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:09.252915 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:09.252919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:09.252922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:09.252945 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:09.255043 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.255512 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.255515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.255517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.255634 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.255926 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.255929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.255931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.256138 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:09.256355 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:09.256513 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 23:22:09.256650 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:22:09.258220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 6074
I20260620 23:22:09.268321 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:09.258819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.0.97.1:44017)
0620 23:22:09.258988 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:09.258996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:09.259304 (+ 308us) server_negotiation.cc:407] Connection header received
0620 23:22:09.259426 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:09.259429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:09.259473 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:09.259538 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:09.260340 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.260915 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.261965 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.262339 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.262428 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:09.264111 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:09.264115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:09.264118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:09.264142 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:09.266327 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.266909 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.266913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.266915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.267021 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.267304 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.267307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.267309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.267534 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:09.267691 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:09.267890 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 23:22:09.268047 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:22:09.268785 6197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:09.347733 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 6074
I20260620 23:22:09.363289 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 6208
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:22:09.377360 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:09.364086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.0.97.2:36409)
0620 23:22:09.364619 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:09.364626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:09.364650 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:22:09.364712 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:09.364718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:09.364923 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:09.365050 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:09.365864 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.366896 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.368046 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.368552 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.368641 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:09.372108 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:09.372119 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:09.372133 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:09.372165 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:09.374850 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.375360 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.375367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.375369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.375475 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.375748 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.375751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.375753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.376030 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:09.376235 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:09.376559 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 23:22:09.376850 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":280,"threads_started":1}
I20260620 23:22:09.378521 6331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:09.454759 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 6208
I20260620 23:22:09.469552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 6341
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:22:09.483624 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:09.470548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.0.97.3:37511)
0620 23:22:09.471154 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:09.471161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:09.471204 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:22:09.471263 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:09.471269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:09.471475 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:09.471597 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:09.472473 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.473421 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.474493 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:09.475162 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:09.475260 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:09.478771 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:09.478780 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:09.478794 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:09.478820 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:09.481224 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.481889 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.481895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.481897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.482001 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:09.482283 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:09.482286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:09.482294 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:09.482576 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:09.482982 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:09.483097 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 23:22:09.483346 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":328,"threads_started":1}
I20260620 23:22:09.484809 6464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:09.563200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 6341
I20260620 23:22:09.580081 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 5981
I20260620 23:22:09.581147 6043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:09.676720 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 5981
2026-06-20T23:22:09Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3668 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[6496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[6496](info): set up 2 sockets
Jun 20 23:22:09 dist-test-slave-5kzz krb5kdc[6496](info): commencing operation
krb5kdc: starting...
W20260620 23:22:10.711108 388 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:22:10 dist-test-slave-5kzz krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 23:22:10.889128 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43331
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:11.188194 6509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:11.188448 6509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:11.188529 6509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:11.195307 6509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:11.195400 6509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:11.195514 6509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:11.195605 6509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:11.195680 6509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:22:11.204835 6509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:11.206058 6509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:11.207330 6509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:11.218806 6514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:11.218852 6517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:11.218876 6515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:11.219429 6509 server_base.cc:1034] running on GCE node
I20260620 23:22:11.220173 6509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:11.221423 6509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:11.222635 6509 hybrid_clock.cc:648] HybridClock initialized: now 1781997731222597 us; error 48 us; skew 500 ppm
I20260620 23:22:11.222975 6509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:11 dist-test-slave-5kzz krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997731, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:11.225752 6509 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 23:22:11.227219 6509 webserver.cc:460] Webserver started at http://127.0.97.62:41105/ using document root <none> and password file <none>
I20260620 23:22:11.227870 6509 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:11.228000 6509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:11.228274 6509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:11.231096 6509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "c302f20b4e83403f9fe29b81eb2f7396"
format_stamp: "Formatted at 2026-06-20 23:22:11 on dist-test-slave-5kzz"
server_key: "596cdbb95caf44c06d92f9da0d2c7934"
server_key_iv: "1061cfb646458ea65d14dba40b470a88"
server_key_version: "encryptionkey@0"
I20260620 23:22:11.231794 6509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "c302f20b4e83403f9fe29b81eb2f7396"
format_stamp: "Formatted at 2026-06-20 23:22:11 on dist-test-slave-5kzz"
server_key: "596cdbb95caf44c06d92f9da0d2c7934"
server_key_iv: "1061cfb646458ea65d14dba40b470a88"
server_key_version: "encryptionkey@0"
I20260620 23:22:11.236406 6509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:22:11.239635 6524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:11.240805 6509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:11.240962 6509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "c302f20b4e83403f9fe29b81eb2f7396"
format_stamp: "Formatted at 2026-06-20 23:22:11 on dist-test-slave-5kzz"
server_key: "596cdbb95caf44c06d92f9da0d2c7934"
server_key_iv: "1061cfb646458ea65d14dba40b470a88"
server_key_version: "encryptionkey@0"
I20260620 23:22:11.241132 6509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:11.257244 6509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:11.260852 6509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:11.261104 6509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:11.278054 6509 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43331
I20260620 23:22:11.278083 6575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43331 every 8 connection(s)
I20260620 23:22:11.279367 6509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:11.283051 6576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:11.284734 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 6509
I20260620 23:22:11.284925 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:22:11.285599 388 external_mini_cluster.cc:1428] Setting key 7346f19376856eea47b8d3f02706531e
I20260620 23:22:11.295616 6576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396: Bootstrap starting.
I20260620 23:22:11.298823 6576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396: Neither blocks nor log segments found. Creating new log.
Jun 20 23:22:11 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:11.299723 6576 log.cc:824] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:11.302644 6576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396: No bootstrap required, opened a new log
I20260620 23:22:11.307484 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:11.287502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.0.97.62:43331)
0620 23:22:11.288270 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:11.288284 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:11.288336 (+ 52us) server_negotiation.cc:407] Connection header received
0620 23:22:11.290570 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:11.290595 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:11.291085 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:11.291520 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:11.292666 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:11.294743 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:11.295852 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:11.296784 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:11.296944 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:11.300591 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:11.300603 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:11.300623 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:11.300677 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:11.303824 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:11.304525 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:11.304530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:11.304540 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:11.304707 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:11.305078 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:11.305081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:11.305082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:11.305513 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:11.305730 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:11.306098 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 23:22:11.306420 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":289,"threads_started":1}
I20260620 23:22:11.311136 6576 raft_consensus.cc:348] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } }
I20260620 23:22:11.311467 6576 raft_consensus.cc:374] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:11.311559 6576 raft_consensus.cc:729] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302f20b4e83403f9fe29b81eb2f7396, State: Initialized, Role: FOLLOWER
I20260620 23:22:11.312114 6576 consensus_queue.cc:260] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } }
I20260620 23:22:11.312336 6576 raft_consensus.cc:388] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:11.312458 6576 raft_consensus.cc:482] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:11.312615 6576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:11.314594 6576 raft_consensus.cc:504] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } }
I20260620 23:22:11.315034 6576 leader_election.cc:302] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c302f20b4e83403f9fe29b81eb2f7396; no voters:
I20260620 23:22:11.315536 6576 leader_election.cc:288] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:22:11.315646 6581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:11.316277 6581 raft_consensus.cc:686] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 1 LEADER]: Becoming Leader. State: Replica: c302f20b4e83403f9fe29b81eb2f7396, State: Running, Role: LEADER
I20260620 23:22:11.316803 6581 consensus_queue.cc:237] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } }
I20260620 23:22:11.317389 6576 sys_catalog.cc:564] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:11.319427 6582 sys_catalog.cc:455] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } }
I20260620 23:22:11.319845 6582 sys_catalog.cc:458] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:11.320353 6583 sys_catalog.cc:455] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c302f20b4e83403f9fe29b81eb2f7396. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } }
I20260620 23:22:11.320693 6583 sys_catalog.cc:458] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:11.322005 6586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:11.328039 6586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:11.336414 6586 catalog_manager.cc:1347] Generated new cluster ID: a322d1c8cca648a09cabb7b34ccf8e82
I20260620 23:22:11.336544 6586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:11.352912 6586 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:22:11.354604 6586 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:11.372816 6586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396: Generated new TSK 0
I20260620 23:22:11.373808 6586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 23:22:11.417973 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43331
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:11.726138 6602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:11.726390 6602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:11.726560 6602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:11.733451 6602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:11.733543 6602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:11.733721 6602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:22:11.743538 6602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:11.744736 6602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:11.746030 6602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:11.758438 6607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:11.758514 6610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:11.758474 6608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:11.759486 6602 server_base.cc:1034] running on GCE node
I20260620 23:22:11.760011 6602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:11.760835 6602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:11.762039 6602 hybrid_clock.cc:648] HybridClock initialized: now 1781997731762000 us; error 49 us; skew 500 ppm
I20260620 23:22:11.762320 6602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:11 dist-test-slave-5kzz krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997731, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:11.765151 6602 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 23:22:11.766438 6602 webserver.cc:460] Webserver started at http://127.0.97.1:43503/ using document root <none> and password file <none>
I20260620 23:22:11.767105 6602 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:11.767199 6602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:11.767431 6602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:11.770018 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "86483dafc79b4aa289bdafbcb739d07e"
format_stamp: "Formatted at 2026-06-20 23:22:11 on dist-test-slave-5kzz"
server_key: "4e9617542434907546f3ba15ba694d5d"
server_key_iv: "a92a9d704316697237c9972e18bc95cc"
server_key_version: "encryptionkey@0"
I20260620 23:22:11.770721 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "86483dafc79b4aa289bdafbcb739d07e"
format_stamp: "Formatted at 2026-06-20 23:22:11 on dist-test-slave-5kzz"
server_key: "4e9617542434907546f3ba15ba694d5d"
server_key_iv: "a92a9d704316697237c9972e18bc95cc"
server_key_version: "encryptionkey@0"
I20260620 23:22:11.775210 6602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:22:11.778254 6617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:11.779556 6602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:22:11.779677 6602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "86483dafc79b4aa289bdafbcb739d07e"
format_stamp: "Formatted at 2026-06-20 23:22:11 on dist-test-slave-5kzz"
server_key: "4e9617542434907546f3ba15ba694d5d"
server_key_iv: "a92a9d704316697237c9972e18bc95cc"
server_key_version: "encryptionkey@0"
I20260620 23:22:11.779845 6602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:11.794629 6602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:11.798270 6602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:11.798484 6602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:11.799364 6602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:11.800527 6602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:11.800599 6602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:11.800674 6602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:11.800719 6602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:11.831667 6602 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40977
I20260620 23:22:11.831709 6729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40977 every 8 connection(s)
I20260620 23:22:11.832850 6602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:22:11.834767 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 6602
I20260620 23:22:11.834956 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:22:11.835688 388 external_mini_cluster.cc:1428] Setting key 64bc3d7e0e1eba5f6cd9903f90436777
Jun 20 23:22:11 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997731, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:11.852780 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:11.835279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42967 (local address 127.0.97.62:43331)
0620 23:22:11.835733 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:11.835742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:11.836829 (+ 1087us) server_negotiation.cc:407] Connection header received
0620 23:22:11.838738 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:11.838747 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:11.838833 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:11.838952 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:11.840330 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:11.841440 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:11.842950 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:11.843387 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:11.843480 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:11.847452 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:11.847456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:11.847460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:11.847487 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:11.849807 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:11.850444 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:11.850452 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:11.850457 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:11.850611 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:11.851154 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:11.851157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:11.851159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:11.851419 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:11.851521 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:11.852391 (+ 870us) server_negotiation.cc:299] Negotiation successful
0620 23:22:11.852547 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":216,"threads_started":1}
I20260620 23:22:11.854324 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:11.835887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.1:42967)
0620 23:22:11.836662 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:11.836718 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:11.838467 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:11.839132 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:11.839139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:11.839609 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:11.840164 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:11.840179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:11.841583 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:11.841587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:11.842824 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:11.842835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:11.843602 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:11.843611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:11.843951 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:11.844702 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:11.844734 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:11.847255 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:11.849937 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:11.849945 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:11.849960 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:11.850337 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:11.850767 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:11.850775 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:11.850781 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:11.851045 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:11.851550 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:11.851560 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:11.852021 (+ 461us) client_negotiation.cc:769] Sending connection context
0620 23:22:11.852348 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 23:22:11.852890 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":283,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:22:11.855948 6730 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43331
I20260620 23:22:11.856241 6730 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:11.856974 6730 heartbeater.cc:511] Master 127.0.97.62:43331 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260620 23:22:11.859084 6541 ts_manager.cc:194] Registered new tserver with Master: 86483dafc79b4aa289bdafbcb739d07e (127.0.97.1:40977)
I20260620 23:22:11.860483 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 23:22:11.877112 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43331
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:12.183259 6736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:12.183514 6736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:12.183653 6736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:12.190984 6736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:12.191103 6736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:12.191263 6736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:22:12.200207 6736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:12.201334 6736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:12.202525 6736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:12.214668 6741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:12.214638 6744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:12.214830 6742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:12.216033 6736 server_base.cc:1034] running on GCE node
I20260620 23:22:12.216571 6736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:12.217556 6736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:12.218770 6736 hybrid_clock.cc:648] HybridClock initialized: now 1781997732218729 us; error 36 us; skew 500 ppm
I20260620 23:22:12.219054 6736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:12 dist-test-slave-5kzz krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997732, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:12.221886 6736 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 23:22:12.223213 6736 webserver.cc:460] Webserver started at http://127.0.97.2:36535/ using document root <none> and password file <none>
I20260620 23:22:12.223829 6736 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:12.223950 6736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:12.224220 6736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:12.226961 6736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "218b5570d52c4d6dab9617e51a26792d"
format_stamp: "Formatted at 2026-06-20 23:22:12 on dist-test-slave-5kzz"
server_key: "58a25efba5636d28c41b662f9c229c45"
server_key_iv: "00ea7ab31673e6adcc6685ec9224e027"
server_key_version: "encryptionkey@0"
I20260620 23:22:12.227690 6736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "218b5570d52c4d6dab9617e51a26792d"
format_stamp: "Formatted at 2026-06-20 23:22:12 on dist-test-slave-5kzz"
server_key: "58a25efba5636d28c41b662f9c229c45"
server_key_iv: "00ea7ab31673e6adcc6685ec9224e027"
server_key_version: "encryptionkey@0"
I20260620 23:22:12.232425 6736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:22:12.235569 6751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:12.236708 6736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:12.236855 6736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "218b5570d52c4d6dab9617e51a26792d"
format_stamp: "Formatted at 2026-06-20 23:22:12 on dist-test-slave-5kzz"
server_key: "58a25efba5636d28c41b662f9c229c45"
server_key_iv: "00ea7ab31673e6adcc6685ec9224e027"
server_key_version: "encryptionkey@0"
I20260620 23:22:12.237027 6736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:12.247093 6736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:12.251003 6736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:12.251264 6736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:12.252091 6736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:12.253291 6736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:12.253393 6736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:12.253518 6736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:12.253587 6736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:12.284061 6736 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42345
I20260620 23:22:12.284154 6863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42345 every 8 connection(s)
I20260620 23:22:12.285259 6736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:22:12.293689 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 6736
I20260620 23:22:12.293848 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:22:12.294332 388 external_mini_cluster.cc:1428] Setting key 728874d18f494702ee314c05b608b66f
Jun 20 23:22:12 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997732, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:12.306277 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:12.287711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56823 (local address 127.0.97.62:43331)
0620 23:22:12.287911 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:12.287917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:12.289022 (+ 1105us) server_negotiation.cc:407] Connection header received
0620 23:22:12.291407 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:12.291411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:12.291472 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:12.291551 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:12.293258 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:12.294141 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:12.296073 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:12.296562 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:12.296724 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:12.300302 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:12.300308 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:12.300312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:12.300340 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:12.303426 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:12.304060 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:12.304068 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:12.304074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:12.304222 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:12.304570 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:12.304577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:12.304583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:12.304914 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:12.305062 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:12.305636 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 23:22:12.305873 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:22:12.306862 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:12.288135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.2:56823)
0620 23:22:12.288868 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:12.288912 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:12.291149 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:12.291771 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:12.291781 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:12.292438 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:12.293106 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:12.293125 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:12.294288 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:12.294291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:12.295873 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:12.295885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:12.296797 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:12.296804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:12.297020 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:12.297665 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:12.297685 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:12.300096 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:12.303555 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:12.303561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:12.303571 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:12.303943 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:12.304319 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:12.304322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:12.304324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:12.304469 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:12.305044 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:12.305050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:12.305297 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 23:22:12.305505 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 23:22:12.305824 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":277,"threads_started":1}
I20260620 23:22:12.308243 6864 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43331
I20260620 23:22:12.308508 6864 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:12.309104 6864 heartbeater.cc:511] Master 127.0.97.62:43331 requested a full tablet report, sending...
I20260620 23:22:12.310375 6541 ts_manager.cc:194] Registered new tserver with Master: 218b5570d52c4d6dab9617e51a26792d (127.0.97.2:42345)
I20260620 23:22:12.311141 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56823
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 23:22:12.335930 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43331
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:12.643814 6869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:12.644052 6869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:12.644136 6869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:12.650583 6869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:12.650714 6869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:12.650851 6869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:22:12.659761 6869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:12.660857 6869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:12.662006 6869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:12.674139 6875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:12.674117 6877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:12.674187 6874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:12.675117 6869 server_base.cc:1034] running on GCE node
I20260620 23:22:12.675714 6869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:12.676545 6869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:12.677757 6869 hybrid_clock.cc:648] HybridClock initialized: now 1781997732677701 us; error 52 us; skew 500 ppm
I20260620 23:22:12.678053 6869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:12 dist-test-slave-5kzz krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997732, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:12.680802 6869 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 23:22:12.682075 6869 webserver.cc:460] Webserver started at http://127.0.97.3:46573/ using document root <none> and password file <none>
I20260620 23:22:12.682732 6869 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:12.682870 6869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:12.683171 6869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:12.685781 6869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "d71f0e04eac14825a32248eaa74d2c86"
format_stamp: "Formatted at 2026-06-20 23:22:12 on dist-test-slave-5kzz"
server_key: "a14baeb5b77849f9369189e846c93e91"
server_key_iv: "15fd15a0cded3c0c32e864bbfa845ee8"
server_key_version: "encryptionkey@0"
I20260620 23:22:12.686460 6869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d71f0e04eac14825a32248eaa74d2c86"
format_stamp: "Formatted at 2026-06-20 23:22:12 on dist-test-slave-5kzz"
server_key: "a14baeb5b77849f9369189e846c93e91"
server_key_iv: "15fd15a0cded3c0c32e864bbfa845ee8"
server_key_version: "encryptionkey@0"
I20260620 23:22:12.691133 6869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260620 23:22:12.694227 6884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:12.695443 6869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:12.695603 6869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "d71f0e04eac14825a32248eaa74d2c86"
format_stamp: "Formatted at 2026-06-20 23:22:12 on dist-test-slave-5kzz"
server_key: "a14baeb5b77849f9369189e846c93e91"
server_key_iv: "15fd15a0cded3c0c32e864bbfa845ee8"
server_key_version: "encryptionkey@0"
I20260620 23:22:12.695789 6869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:12.707922 6869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:12.711510 6869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:12.711807 6869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:12.712638 6869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:12.713863 6869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:12.713954 6869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:12.714082 6869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:12.714154 6869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:12.749495 6869 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38711
I20260620 23:22:12.749539 6996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38711 every 8 connection(s)
I20260620 23:22:12.750628 6869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:22:12.751426 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 6869
I20260620 23:22:12.751585 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:22:12.752123 388 external_mini_cluster.cc:1428] Setting key 8b61849f9d5263d31cbba3c26ce314bb
Jun 20 23:22:12 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997732, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:12.768911 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:12.753299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49985 (local address 127.0.97.62:43331)
0620 23:22:12.753509 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:12.753516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:12.754738 (+ 1222us) server_negotiation.cc:407] Connection header received
0620 23:22:12.756447 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:12.756450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:12.756513 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:12.756592 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:12.757921 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:12.758596 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:12.760156 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:12.760551 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:12.760641 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:12.763608 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:12.763612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:12.763615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:12.763641 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:12.765967 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:12.766782 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:12.766787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:12.766789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:12.766890 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:12.767358 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:12.767363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:12.767366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:12.767609 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:12.767772 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:12.768417 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 23:22:12.768593 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:22:12.769670 6999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:12.753797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.3:49985)
0620 23:22:12.754565 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:12.754608 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:12.756216 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:12.756785 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:12.756793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:12.757259 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:12.757800 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:12.757816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:12.758793 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:12.758799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:12.760037 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:12.760048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:12.760710 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:12.760714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:12.760919 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:12.761470 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:12.761487 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:12.763449 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:12.766099 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:12.766106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:12.766118 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:12.766582 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:12.766996 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:12.767002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:12.767005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:12.767246 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:12.767734 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:12.767744 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:12.768082 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 23:22:12.768308 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 23:22:12.768726 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":324,"threads_started":1}
I20260620 23:22:12.770887 6997 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43331
I20260620 23:22:12.771154 6997 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:12.771776 6997 heartbeater.cc:511] Master 127.0.97.62:43331 requested a full tablet report, sending...
I20260620 23:22:12.772969 6541 ts_manager.cc:194] Registered new tserver with Master: d71f0e04eac14825a32248eaa74d2c86 (127.0.97.3:38711)
I20260620 23:22:12.773656 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49985
I20260620 23:22:12.778900 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260620 23:22:12.819634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45803
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43331,127.0.97.61:45803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:22:12.863678 6730 heartbeater.cc:503] Master 127.0.97.62:43331 was elected leader, sending a full tablet report...
W20260620 23:22:13.125205 7002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:13.125512 7002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:13.125684 7002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:13.132715 7002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:13.132819 7002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:13.132867 7002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:13.132946 7002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:13.133031 7002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:22:13.142517 7002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43331,127.0.97.61:45803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
W20260620 23:22:13.142871 7002 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:43331,127.0.97.61:45803'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 23:22:13.143831 7002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:13.145244 7002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:13.156477 7007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:13.156513 7008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:13.156759 7010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:13.157446 7002 server_base.cc:1034] running on GCE node
I20260620 23:22:13.158169 7002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:13.159466 7002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:13.160722 7002 hybrid_clock.cc:648] HybridClock initialized: now 1781997733160653 us; error 43 us; skew 500 ppm
I20260620 23:22:13.161042 7002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997733, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:13.164099 7002 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 23:22:13.165483 7002 webserver.cc:460] Webserver started at http://127.0.97.61:44475/ using document root <none> and password file <none>
I20260620 23:22:13.166141 7002 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:13.166265 7002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:13.166571 7002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:13.169549 7002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data/instance:
uuid: "d3316daf2d0b47188882ae176b98bf12"
format_stamp: "Formatted at 2026-06-20 23:22:13 on dist-test-slave-5kzz"
server_key: "77a32418e1ab4648aba0f820168a3210"
server_key_iv: "e50a441e70257b725f675f5ae1bca1c0"
server_key_version: "encryptionkey@0"
I20260620 23:22:13.170322 7002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/wal/instance:
uuid: "d3316daf2d0b47188882ae176b98bf12"
format_stamp: "Formatted at 2026-06-20 23:22:13 on dist-test-slave-5kzz"
server_key: "77a32418e1ab4648aba0f820168a3210"
server_key_iv: "e50a441e70257b725f675f5ae1bca1c0"
server_key_version: "encryptionkey@0"
I20260620 23:22:13.175277 7002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:22:13.178496 7017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:13.179787 7002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:22:13.179982 7002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/wal
uuid: "d3316daf2d0b47188882ae176b98bf12"
format_stamp: "Formatted at 2026-06-20 23:22:13 on dist-test-slave-5kzz"
server_key: "77a32418e1ab4648aba0f820168a3210"
server_key_iv: "e50a441e70257b725f675f5ae1bca1c0"
server_key_version: "encryptionkey@0"
I20260620 23:22:13.180179 7002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:13.199649 7002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:13.203717 7002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:13.203915 7002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:13.220728 7002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45803
I20260620 23:22:13.220743 7068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45803 every 8 connection(s)
I20260620 23:22:13.221966 7002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
I20260620 23:22:13.225479 7069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:13.225925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 7002
I20260620 23:22:13.233551 7069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997733, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:13.247690 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.227860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.0.97.61:45803)
0620 23:22:13.228628 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.228638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.228674 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:22:13.230729 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.230751 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.231118 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.231417 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.233096 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.234401 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.236003 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.237078 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.237244 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.241098 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.241108 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.241123 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.241196 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.244118 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.244728 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.244736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.244744 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.244890 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.245278 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.245285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.245289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.245708 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.246009 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.246374 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.246794 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":355,"threads_started":1}
I20260620 23:22:13.249531 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.236091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.0.97.62:43331)
0620 23:22:13.236244 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.236248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.236903 (+ 655us) server_negotiation.cc:407] Connection header received
0620 23:22:13.237090 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.237095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.237162 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.237258 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.238255 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.238973 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.240270 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.240744 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.240865 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.244066 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.244072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.244077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.244113 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.246966 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.247618 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.247623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.247626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.247747 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.248177 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.248184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.248188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.248504 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.248640 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.249042 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.249230 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:22:13.249646 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.236109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53166)
0620 23:22:13.236773 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.236810 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.236970 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.237308 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.237316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.237894 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.238121 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.238129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.239106 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.239111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.240147 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.240158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.240947 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.240954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.241164 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.241783 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.241798 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.243865 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:13.247089 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.247099 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.247105 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.247502 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.247965 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.247968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.247970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.248081 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.248619 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:13.248626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:13.248771 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.248953 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.249226 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"spinlock_wait_cycles":11008,"thread_start_us":333,"threads_started":1}
I20260620 23:22:13.251133 7069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997733, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 23:22:13.261505 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.251695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45803 (local address 127.0.0.1:59646)
0620 23:22:13.251849 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.251862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.251955 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.252189 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.252193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.252511 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.252645 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.252653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.253533 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.253540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.254380 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.254390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.255056 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.255062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.255268 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.255675 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.255688 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.257321 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:13.259843 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.259846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.259848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.260150 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.260419 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.260421 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.260424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.260533 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.260947 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:13.260950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:13.261047 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.261137 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.261292 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":3}
I20260620 23:22:13.261646 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.251842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.0.97.61:45803)
0620 23:22:13.251955 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.251960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.251974 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:22:13.252047 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.252051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.252096 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.252177 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.252763 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.253416 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.254511 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.254931 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.255052 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.257464 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.257469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.257472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.257497 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.259744 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.260236 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.260239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.260241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.260338 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.260609 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.260612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.260614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.260851 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.260958 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.261202 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.261377 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":38}
I20260620 23:22:13.265690 7069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: Bootstrap starting.
I20260620 23:22:13.268623 7069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:13.269501 7069 log.cc:824] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:13.271761 7069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: No bootstrap required, opened a new log
I20260620 23:22:13.278152 7069 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } }
I20260620 23:22:13.278441 7069 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:13.278533 7069 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3316daf2d0b47188882ae176b98bf12, State: Initialized, Role: FOLLOWER
I20260620 23:22:13.279117 7069 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } }
I20260620 23:22:13.279930 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } } }
I20260620 23:22:13.280249 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:22:13.280946 7069 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:22:13.289968 7086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:22:13.290120 7086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:22:13.292181 7002 master_runner.cc:428] Detected that this master d3316daf2d0b47188882ae176b98bf12 is joining an existing cluster
I20260620 23:22:13.292268 7002 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:45803
I20260620 23:22:13.311506 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.295916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.0.97.62:43331)
0620 23:22:13.296100 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.296105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.296863 (+ 758us) server_negotiation.cc:407] Connection header received
0620 23:22:13.297124 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.297132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.297208 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.297274 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.298803 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.299868 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.301878 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.302472 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.302562 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.305006 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.305012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.305017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.305053 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.308335 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.309173 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.309179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.309183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.309332 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.309785 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.309790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.309794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.310154 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.310267 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.310919 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.311134 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:22:13.312819 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.296090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.0.97.61:45803)
0620 23:22:13.296556 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.296564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.297383 (+ 819us) server_negotiation.cc:407] Connection header received
0620 23:22:13.297550 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.297555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.297611 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.297710 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.298772 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.299544 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.301298 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.301979 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.302067 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.304973 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.304981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.304988 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.305022 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.309778 (+ 4756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.310593 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.310598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.310602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.310767 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.311224 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.311231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.311237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.311599 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.311737 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.312234 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.312449 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":288}
I20260620 23:22:13.313428 6864 heartbeater.cc:503] Master 127.0.97.62:43331 was elected leader, sending a full tablet report...
I20260620 23:22:13.313616 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.296099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53182)
0620 23:22:13.296755 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.296775 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.296966 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.297458 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.297464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.298200 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.298616 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.298628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.300044 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.300049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.301698 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.301711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.302860 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.302866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.303021 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.303600 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.303621 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.304824 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:13.308476 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.308482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.308486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.309025 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.309471 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.309476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.309479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.309651 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.310359 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:13.310364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:13.310566 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.312026 (+ 1460us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.313197 (+ 1171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":29,"spinlock_wait_cycles":11648,"thread_start_us":480,"threads_started":1}
I20260620 23:22:13.315981 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.296115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45803 (local address 127.0.0.1:59652)
0620 23:22:13.297289 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.297306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.297438 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.297721 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.297724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.298254 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.298618 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.298627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.299758 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.299761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.301162 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.301174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.302364 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.302370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.302542 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.303273 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.303551 (+ 278us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.304809 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:13.309918 (+ 5109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.309929 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.309935 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.310459 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.310927 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.310932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.310938 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.311101 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.311818 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:13.311825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:13.311980 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.312341 (+ 361us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.312634 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"mutex_wait_us":24,"spinlock_wait_cycles":22784,"thread_start_us":914,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997731, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 23:22:13.336251 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.320105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.0.97.61:45803)
0620 23:22:13.320845 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.320852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.320872 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:22:13.320962 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.320969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.321106 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.321211 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.322504 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.323841 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.325654 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.326138 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.326229 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.330100 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.330108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.330115 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.330155 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.333440 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.334113 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.334120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.334128 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.334272 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.334710 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.334715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.334719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.335054 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.335195 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.335694 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.335913 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599}
I20260620 23:22:13.336452 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.319782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45803 (local address 127.0.0.1:59666)
0620 23:22:13.320387 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.320433 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.320585 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.321274 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.321281 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.322065 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.322350 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.322362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.323979 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.323987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.325519 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.325533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.326290 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.326296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.326499 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.327091 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.327120 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.329917 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:13.333571 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.333581 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.333588 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.334000 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.334387 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.334394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.334401 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.334569 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.335179 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:13.335189 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:13.335402 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.335595 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.335943 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"spinlock_wait_cycles":768,"thread_start_us":362,"threads_started":1}
I20260620 23:22:13.338507 6540 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:45803: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d3316daf2d0b47188882ae176b98bf12"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 45803
}
attrs {
promote: true
}
}
dest_uuid: "c302f20b4e83403f9fe29b81eb2f7396"
cas_config_opid_index: -1
I20260620 23:22:13.339668 6540 consensus_queue.cc:237] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true } }
I20260620 23:22:13.348513 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d3316daf2d0b47188882ae176b98bf12 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true } } }
I20260620 23:22:13.349529 7044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:13.351998 7098 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:45803
I20260620 23:22:13.352389 7098 sys_catalog.cc:455] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true } } }
I20260620 23:22:13.352669 7002 master.cc:537] Master@127.0.97.61:45803 shutting down...
I20260620 23:22:13.352718 7098 sys_catalog.cc:458] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:13.354559 7044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 FOLLOWER]: Refusing update from remote peer c302f20b4e83403f9fe29b81eb2f7396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 23:22:13.357429 7098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260620 23:22:13.360565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37761
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43331,127.0.97.61:45803,127.0.97.60:37761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:13.361446 6525 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 23:22:13.364205 6525 consensus_peers.cc:487] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 -> Peer d3316daf2d0b47188882ae176b98bf12 (127.0.97.61:45803): Couldn't send request to peer d3316daf2d0b47188882ae176b98bf12. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 23:22:13.366212 7002 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:13.366550 7002 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:13.366796 7002 tablet_replica.cc:330] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: stopping tablet replica
I20260620 23:22:13.373276 7002 master.cc:552] Master@127.0.97.61:45803 shutdown complete.
I20260620 23:22:13.373353 7002 master_runner.cc:305] Clearing existing system tablet
I20260620 23:22:13.373755 7002 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:13.373848 7002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:13.376645 7002 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:22:13.378252 7104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:13.378647 7002 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:13.378944 7002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:22:13.379051 7002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/wal
uuid: "d3316daf2d0b47188882ae176b98bf12"
format_stamp: "Formatted at 2026-06-20 23:22:13 on dist-test-slave-5kzz"
server_key: "77a32418e1ab4648aba0f820168a3210"
server_key_iv: "e50a441e70257b725f675f5ae1bca1c0"
server_key_version: "encryptionkey@0"
I20260620 23:22:13.383992 7002 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:13.390586 7002 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:22:13.390756 7002 log.cc:1196] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 23:22:13.391178 7002 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: Deleting consensus metadata
I20260620 23:22:13.391420 7002 master_runner.cc:315] Copying system tablet from 127.0.97.62:43331
I20260620 23:22:13.399964 7002 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:43331
I20260620 23:22:13.415409 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.400736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.0.97.62:43331)
0620 23:22:13.400996 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.401002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.401446 (+ 444us) server_negotiation.cc:407] Connection header received
0620 23:22:13.401620 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.401626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.401721 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.401852 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.402781 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.403814 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.405657 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.406349 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.406479 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.408629 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.408634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.408638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.408673 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.412440 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.413231 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.413238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.413243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.413407 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.413849 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.413856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.413862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.414266 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.414439 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.414864 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.415071 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:22:13.415556 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.400671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53186)
0620 23:22:13.401317 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.401336 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.401485 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.401885 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.401889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.402351 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.402611 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.402623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.403977 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.403982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.405475 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.405491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.406503 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.406510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.406707 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.407414 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.407437 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.408431 (+ 994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:13.412628 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.412633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.412638 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.413098 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.413550 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.413555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.413559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.413721 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.414396 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:13.414401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:13.414561 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.414769 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.415094 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":431,"threads_started":1}
I20260620 23:22:13.416486 6561 tablet_copy_service.cc:140] P c302f20b4e83403f9fe29b81eb2f7396: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d3316daf2d0b47188882ae176b98bf12 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:53186)
I20260620 23:22:13.416749 6561 tablet_copy_service.cc:161] P c302f20b4e83403f9fe29b81eb2f7396: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d3316daf2d0b47188882ae176b98bf12 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:53186: session id = d3316daf2d0b47188882ae176b98bf12-00000000000000000000000000000000
I20260620 23:22:13.421136 6561 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:22:13.423233 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:22:13.429776 7002 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: tablet copy: Starting download of 0 data blocks...
I20260620 23:22:13.430090 7002 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: tablet copy: Starting download of 1 WAL segments...
I20260620 23:22:13.432696 7002 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:22:13.436872 6561 tablet_copy_service.cc:342] P c302f20b4e83403f9fe29b81eb2f7396: Request end of tablet copy session d3316daf2d0b47188882ae176b98bf12-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:53186
I20260620 23:22:13.437062 6561 tablet_copy_service.cc:433] P c302f20b4e83403f9fe29b81eb2f7396: ending tablet copy session d3316daf2d0b47188882ae176b98bf12-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d3316daf2d0b47188882ae176b98bf12
I20260620 23:22:13.448563 7002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:13.451092 7111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:13.452389 7112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:13.462539 7002 server_base.cc:1034] running on GCE node
W20260620 23:22:13.462919 7114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:13.463310 7002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:13.463927 7002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:13.465178 7002 hybrid_clock.cc:648] HybridClock initialized: now 1781997733465121 us; error 31 us; skew 500 ppm
I20260620 23:22:13.465385 7002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997733, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:13.467540 7002 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 23:22:13.468858 7002 webserver.cc:460] Webserver started at http://127.0.97.61:41843/ using document root <none> and password file <none>
I20260620 23:22:13.469189 7002 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:13.469270 7002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:13.472234 7002 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:13.474102 7121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:13.474500 7002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:22:13.474628 7002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/wal
uuid: "d3316daf2d0b47188882ae176b98bf12"
format_stamp: "Formatted at 2026-06-20 23:22:13 on dist-test-slave-5kzz"
server_key: "77a32418e1ab4648aba0f820168a3210"
server_key_iv: "e50a441e70257b725f675f5ae1bca1c0"
server_key_version: "encryptionkey@0"
I20260620 23:22:13.474818 7002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:13.487864 7002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:13.491669 7002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:13.519647 7002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45803
I20260620 23:22:13.519731 7172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45803 every 8 connection(s)
I20260620 23:22:13.520709 7002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
I20260620 23:22:13.526026 7173 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:22:13.529755 7173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: Bootstrap starting.
I20260620 23:22:13.546859 7173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:22:13.547499 7173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: Bootstrap complete.
I20260620 23:22:13.548892 7173 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true } }
I20260620 23:22:13.549221 7173 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3316daf2d0b47188882ae176b98bf12, State: Initialized, Role: LEARNER
I20260620 23:22:13.549510 7173 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true } }
I20260620 23:22:13.550417 7176 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true } } }
I20260620 23:22:13.550697 7173 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:13.550813 7176 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:22:13.557983 7187 catalog_manager.cc:1259] Loaded cluster ID: a322d1c8cca648a09cabb7b34ccf8e82
I20260620 23:22:13.558115 7187 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: loading cluster ID for follower catalog manager: success
I20260620 23:22:13.561215 7187 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: acquiring CA information for follower catalog manager: success
I20260620 23:22:13.563378 7187 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 23:22:13.725217 7102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:13.725510 7102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:13.725651 7102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:13.733099 7102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:13.733232 7102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:13.733322 7102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:13.733433 7102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:13.733503 7102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:22:13.743386 7102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43331,127.0.97.61:45803,127.0.97.60:37761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:13.744760 7102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:13.746021 7102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:13.758163 7195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:13.758139 7193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:13.758193 7192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:13.758953 7102 server_base.cc:1034] running on GCE node
I20260620 23:22:13.759644 7102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:13.760845 7102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:13.762049 7102 hybrid_clock.cc:648] HybridClock initialized: now 1781997733761993 us; error 46 us; skew 500 ppm
I20260620 23:22:13.762372 7102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997733, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:13.765162 7102 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 23:22:13.766431 7102 webserver.cc:460] Webserver started at http://127.0.97.60:34175/ using document root <none> and password file <none>
I20260620 23:22:13.767207 7102 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:13.767346 7102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:13.767629 7102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:13.770740 7102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data/instance:
uuid: "df8d3de65ed749678f8a3fb4c3d20588"
format_stamp: "Formatted at 2026-06-20 23:22:13 on dist-test-slave-5kzz"
server_key: "4b11313a853fc05d58e15ca40cd45d7d"
server_key_iv: "a647b9df5aa79345a41177f5838762ff"
server_key_version: "encryptionkey@0"
I20260620 23:22:13.771493 7102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/wal/instance:
uuid: "df8d3de65ed749678f8a3fb4c3d20588"
format_stamp: "Formatted at 2026-06-20 23:22:13 on dist-test-slave-5kzz"
server_key: "4b11313a853fc05d58e15ca40cd45d7d"
server_key_iv: "a647b9df5aa79345a41177f5838762ff"
server_key_version: "encryptionkey@0"
I20260620 23:22:13.775776 6997 heartbeater.cc:503] Master 127.0.97.62:43331 was elected leader, sending a full tablet report...
I20260620 23:22:13.776350 7102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:22:13.779524 7202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:13.780975 7102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:22:13.781124 7102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/wal
uuid: "df8d3de65ed749678f8a3fb4c3d20588"
format_stamp: "Formatted at 2026-06-20 23:22:13 on dist-test-slave-5kzz"
server_key: "4b11313a853fc05d58e15ca40cd45d7d"
server_key_iv: "a647b9df5aa79345a41177f5838762ff"
server_key_version: "encryptionkey@0"
I20260620 23:22:13.781306 7102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:13.804945 7102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:13.808853 7102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:13.809094 7102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:13.826958 7102 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37761
I20260620 23:22:13.826979 7253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37761 every 8 connection(s)
I20260620 23:22:13.828220 7102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
I20260620 23:22:13.832294 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:13.833604 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 7102
I20260620 23:22:13.841346 7254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997733, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:13.856242 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.835649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.0.97.60:37761)
0620 23:22:13.836427 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.836438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.836477 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:22:13.838195 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.838215 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.838601 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.839007 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.840810 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.842179 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.843830 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.845157 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.845340 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.849485 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.849492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.849501 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.849531 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.852547 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.853194 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.853199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.853204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.853331 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.853724 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.853729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.853732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.854256 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.854542 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.854974 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.855317 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":362,"threads_started":1}
I20260620 23:22:13.858028 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.844057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.0.97.62:43331)
0620 23:22:13.844235 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.844241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.845067 (+ 826us) server_negotiation.cc:407] Connection header received
0620 23:22:13.845322 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.845326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.845386 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.845497 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.846658 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.847406 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.849011 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.849402 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.849531 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.852838 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.852844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.852849 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.852885 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.855540 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.856203 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.856211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.856215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.856375 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.856766 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.856771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.856774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.857097 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.857239 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.857668 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.857820 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:22:13.858292 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.843867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53192)
0620 23:22:13.844931 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.844988 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.845170 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.845581 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.845587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.846284 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.846519 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.846527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.847536 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.847541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.848881 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.848891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.849571 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.849577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.849753 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.850424 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.850458 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.852670 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:13.855665 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.855674 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.855680 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.856091 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.856501 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.856505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.856509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.856655 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.857219 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:13.857224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:13.857357 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.857589 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.857899 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":261,"threads_started":1}
I20260620 23:22:13.860077 7254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997733, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 23:22:13.872673 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.860709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45803 (local address 127.0.0.1:59672)
0620 23:22:13.860996 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.861018 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.861113 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.861517 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.861521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.861845 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.861953 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.861960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.863209 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.863215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.864075 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.864085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.864763 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.864769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.864961 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.865450 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.865465 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.867572 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:13.870375 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.870382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.870386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.870800 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.871177 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.871183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.871187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.871314 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.871772 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:13.871777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:13.871916 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.872043 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.872307 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260620 23:22:13.872707 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.860758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.0.97.61:45803)
0620 23:22:13.861265 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.861270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.861284 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:22:13.861330 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.861333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.861392 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.861503 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.862091 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.863087 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.864209 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.864611 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.864744 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.867744 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.867749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.867752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.867779 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.870250 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.870907 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.870914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.870918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.871043 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.871418 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.871425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.871428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.871660 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.871814 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.872134 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.872332 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":253,"threads_started":1}
I20260620 23:22:13.873662 7254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997733, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 23:22:13.884575 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.874204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37761 (local address 127.0.0.1:45794)
0620 23:22:13.874378 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.874391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.874491 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.874783 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.874787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.875115 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.875226 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.875233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.876112 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.876120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.876914 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.876923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.877555 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.877560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.877749 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.878181 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.878196 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.879963 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:13.882531 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.882534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.882536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.882915 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.883201 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.883204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.883206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.883369 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.883803 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:13.883820 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:13.883942 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.884041 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.884198 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:22:13.884584 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.874385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.0.97.60:37761)
0620 23:22:13.874555 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.874561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.874577 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:22:13.874615 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.874617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.874659 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.874770 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.875352 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.876003 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.877036 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.877415 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.877543 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.880113 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.880117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.880121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.880146 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.882431 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.883010 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.883014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.883016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.883119 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.883495 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.883498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.883500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.883717 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.883845 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.884124 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.884289 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":51}
I20260620 23:22:13.889127 7254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: Bootstrap starting.
I20260620 23:22:13.892125 7254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:13.893085 7254 log.cc:824] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:13.895524 7254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: No bootstrap required, opened a new log
I20260620 23:22:13.903010 7254 raft_consensus.cc:348] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } }
I20260620 23:22:13.903299 7254 raft_consensus.cc:374] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:13.903419 7254 raft_consensus.cc:729] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8d3de65ed749678f8a3fb4c3d20588, State: Initialized, Role: FOLLOWER
I20260620 23:22:13.903959 7254 consensus_queue.cc:260] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } }
I20260620 23:22:13.904706 7261 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } } }
I20260620 23:22:13.904992 7261 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:22:13.905637 7254 sys_catalog.cc:564] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:13.911428 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 6602
W20260620 23:22:13.915366 7272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:22:13.915548 7272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:22:13.920122 7102 master_runner.cc:428] Detected that this master df8d3de65ed749678f8a3fb4c3d20588 is joining an existing cluster
I20260620 23:22:13.920226 7102 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:37761
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 23:22:13.936275 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.912417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.0.97.1:40977)
0620 23:22:13.916114 (+ 3697us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.916123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.916148 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:22:13.916210 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.916217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.916444 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.916653 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.918074 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.919683 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.921559 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.922488 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.922615 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.928715 (+ 6100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.928727 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.928744 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.928785 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.932446 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.933260 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.933269 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.933272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.933421 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.933890 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.933895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.933899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.934303 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.935489 (+ 1186us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.935639 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.935886 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3539,"thread_start_us":3375,"threads_started":1}
I20260620 23:22:13.938057 6725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:13.959478 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.924167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.0.97.62:43331)
0620 23:22:13.924315 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.924319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.940003 (+ 15684us) server_negotiation.cc:407] Connection header received
0620 23:22:13.940614 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.940622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.940709 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.941183 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.942715 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.943887 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.946369 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.947010 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.947099 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.949648 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.949654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.949659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.949695 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.953643 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.954339 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.954346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.954350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.954479 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.957814 (+ 3335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.957819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.957823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.958151 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.958261 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.958887 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.959106 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:22:13.960495 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.924755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53200)
0620 23:22:13.939875 (+ 15120us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.939904 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.940416 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.941406 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.941413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.942143 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.942517 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.942530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.944108 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.944113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.946230 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.946242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.947375 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.947381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.947566 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.948444 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.948463 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.949471 (+ 1008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:13.953802 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.953807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.953811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.954226 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.954610 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.954615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.954619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.954804 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.958355 (+ 3551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:13.958360 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:13.958571 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.959715 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.960036 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14948,"thread_start_us":15268,"threads_started":1}
I20260620 23:22:13.968382 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.924308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.0.97.61:45803)
0620 23:22:13.924499 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.924505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.942611 (+ 18106us) server_negotiation.cc:407] Connection header received
0620 23:22:13.943289 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.943294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.943398 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.943483 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.946032 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.947075 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.949272 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.949949 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.950039 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.954247 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.954255 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.954263 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.954305 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.957627 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.965060 (+ 7433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.965067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.965072 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.965232 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.966764 (+ 1532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.966769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.966772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.967095 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.967221 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.967805 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.968060 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:22:13.969570 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.924792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45803 (local address 127.0.0.1:59676)
0620 23:22:13.942517 (+ 17725us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.942533 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.943046 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.943671 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.943675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.944369 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.944690 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.944700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.947302 (+ 2602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.947307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.949004 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.949016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.950370 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.950381 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.950600 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.951979 (+ 1379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.952004 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.953994 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:13.961732 (+ 7738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.961738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.961741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.962159 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.965722 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.965729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.965736 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.966463 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.967312 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:13.967317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:13.967546 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.967684 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.969099 (+ 1415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17562,"thread_start_us":16356,"threads_started":1}
I20260620 23:22:13.981307 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.941697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.0.97.60:37761)
0620 23:22:13.941897 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.941905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.948309 (+ 6404us) server_negotiation.cc:407] Connection header received
0620 23:22:13.960900 (+ 12591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.960907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.960971 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.961049 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.968069 (+ 7020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.969284 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.971783 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.972567 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.972667 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.975227 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.975233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.975237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.975270 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:13.978711 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.979399 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.979404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.979408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.979515 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:13.979831 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:13.979847 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:13.979850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:13.980161 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:13.980292 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.980775 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.980984 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:22:13.982491 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.942004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37761 (local address 127.0.0.1:45796)
0620 23:22:13.948214 (+ 6210us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.948233 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.960754 (+ 12521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.961220 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.961224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.966579 (+ 5355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.967937 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.967947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.969529 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.969536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.971643 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.971659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.972999 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.973006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.973214 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.973917 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.973944 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.975015 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:13.978853 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.978858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.978861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.979292 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.979621 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:13.979625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:13.979629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:13.979743 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:13.980350 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:13.980354 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:13.980545 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.981769 (+ 1224us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.982045 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6089,"thread_start_us":7091,"threads_started":1}
Jun 20 23:22:13 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997731, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 23:22:13.997750 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.986498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59678 (local address 127.0.97.61:45803)
0620 23:22:13.986647 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.986653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.989884 (+ 3231us) server_negotiation.cc:407] Connection header received
0620 23:22:13.990058 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.990063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.990144 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.990253 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:13.990967 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.992497 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.995063 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.996367 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.996478 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.996730 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:13.997350 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 23:22:13.997487 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:22:13.999176 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.986408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45803 (local address 127.0.0.1:59678)
0620 23:22:13.989779 (+ 3371us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.989792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.989929 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.990416 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.990421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.990453 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:13.990787 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.990798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.992674 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.992679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.994890 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.994904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.996935 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.996941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.997115 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.997185 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 23:22:13.998607 (+ 1422us) client_negotiation.cc:240] Negotiation successful
0620 23:22:13.998760 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3271,"thread_start_us":3143,"threads_started":1}
I20260620 23:22:14.000460 7148 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 23:22:14.001240 7176 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c302f20b4e83403f9fe29b81eb2f7396. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true } } }
I20260620 23:22:14.001466 7176 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:22:14.002804 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.967412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.0.97.60:37761)
0620 23:22:13.981737 (+ 14325us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:13.981743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:13.985991 (+ 4248us) server_negotiation.cc:407] Connection header received
0620 23:22:13.986878 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:13.986884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:13.986962 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:13.987055 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:13.988108 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.989142 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.991031 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.992149 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.992235 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.996187 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:13.996193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:13.996198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:13.996230 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:14.000038 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:14.000770 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:14.000775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:14.000779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:14.000906 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:14.001266 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:14.001271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:14.001275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:14.001560 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:14.001682 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:14.002226 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 23:22:14.002431 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14227,"thread_start_us":21919,"threads_started":1}
I20260620 23:22:14.003405 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:13.967351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37761 (local address 127.0.0.1:45800)
0620 23:22:13.985891 (+ 18540us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:13.985914 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:13.986627 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:13.987234 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:13.987240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:13.987724 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:13.987967 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.987977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.989304 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.989309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.990876 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:13.990890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:13.992463 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:13.992469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:13.992641 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:13.993309 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:13.993332 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:13.996002 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:14.000195 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:14.000200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:14.000204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:14.000652 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:14.001031 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:14.001035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:14.001039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:14.001165 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:14.001789 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:14.001794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:14.001995 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 23:22:14.002754 (+ 759us) client_negotiation.cc:240] Negotiation successful
0620 23:22:14.003026 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18415,"thread_start_us":18666,"threads_started":1}
I20260620 23:22:14.004812 6540 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:37761: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 37761
}
attrs {
promote: true
}
}
dest_uuid: "c302f20b4e83403f9fe29b81eb2f7396"
cas_config_opid_index: 5
I20260620 23:22:14.005718 6540 consensus_queue.cc:237] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } }
I20260620 23:22:14.007629 7283 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER df8d3de65ed749678f8a3fb4c3d20588 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } } }
I20260620 23:22:14.008443 7229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:14.011101 7148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 LEARNER]: Refusing update from remote peer c302f20b4e83403f9fe29b81eb2f7396: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 23:22:14.011972 7229 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 FOLLOWER]: Refusing update from remote peer c302f20b4e83403f9fe29b81eb2f7396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 23:22:14.015044 7287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 23:22:14.015966 7283 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:37761
I20260620 23:22:14.016566 7102 master.cc:537] Master@127.0.97.60:37761 shutting down...
I20260620 23:22:14.016067 7288 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.002s
I20260620 23:22:14.016587 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } } }
I20260620 23:22:14.018198 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:14.037252 7148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER df8d3de65ed749678f8a3fb4c3d20588 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } } }
I20260620 23:22:14.041618 7176 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: true } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } } }
I20260620 23:22:14.041900 7176 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:22:14.042789 7289 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396: attempting to promote NON_VOTER d3316daf2d0b47188882ae176b98bf12 to VOTER
I20260620 23:22:14.043489 7289 consensus_queue.cc:237] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } }
I20260620 23:22:14.044654 7102 raft_consensus.cc:2229] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:14.045027 7102 raft_consensus.cc:2258] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:14.045236 7102 tablet_replica.cc:330] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: stopping tablet replica
I20260620 23:22:14.046916 7148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 LEARNER]: Refusing update from remote peer c302f20b4e83403f9fe29b81eb2f7396: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 23:22:14.047389 7288 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 23:22:14.049139 6525 consensus_peers.cc:487] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 -> Peer df8d3de65ed749678f8a3fb4c3d20588 (127.0.97.60:37761): Couldn't send request to peer df8d3de65ed749678f8a3fb4c3d20588. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:22:14.051878 7102 master.cc:552] Master@127.0.97.60:37761 shutdown complete.
I20260620 23:22:14.051961 7102 master_runner.cc:305] Clearing existing system tablet
I20260620 23:22:14.052347 7102 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:14.052460 7102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:14.055333 7102 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 23:22:14.055480 7287 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d3316daf2d0b47188882ae176b98bf12 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } } }
I20260620 23:22:14.059930 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } } }
I20260620 23:22:14.060268 7288 sys_catalog.cc:455] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } } }
I20260620 23:22:14.060503 7288 sys_catalog.cc:458] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:14.060698 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:14.061403 7148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d3316daf2d0b47188882ae176b98bf12 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } } }
I20260620 23:22:14.065336 7176 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } } }
I20260620 23:22:14.065551 7176 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:22:14.067078 7294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:14.067427 7102 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:14.067682 7102 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.004s
I20260620 23:22:14.067813 7102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/wal
uuid: "df8d3de65ed749678f8a3fb4c3d20588"
format_stamp: "Formatted at 2026-06-20 23:22:13 on dist-test-slave-5kzz"
server_key: "4b11313a853fc05d58e15ca40cd45d7d"
server_key_iv: "a647b9df5aa79345a41177f5838762ff"
server_key_version: "encryptionkey@0"
I20260620 23:22:14.072666 7102 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:14.079144 7102 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:22:14.079286 7102 log.cc:1196] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 23:22:14.079696 7102 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: Deleting consensus metadata
I20260620 23:22:14.079933 7102 master_runner.cc:315] Copying system tablet from 127.0.97.62:43331
I20260620 23:22:14.084398 7102 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:43331
I20260620 23:22:14.096215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 6602
I20260620 23:22:14.103883 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:14.085155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53212)
0620 23:22:14.085705 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:14.085727 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:14.085874 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:14.086250 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:14.086254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:14.086812 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:14.087531 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:14.087544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.089047 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.089052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:14.090716 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:14.090735 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.091725 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.091733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:14.092006 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:14.096023 (+ 4017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:14.096047 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:14.097120 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:14.100960 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:14.100968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:14.100975 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:14.101499 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:14.101884 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:14.101891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:14.101897 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:14.102057 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:14.102781 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:14.102786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:14.102957 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 23:22:14.103112 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 23:22:14.103414 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":305,"threads_started":1}
I20260620 23:22:14.103940 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:14.085166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.0.97.62:43331)
0620 23:22:14.085367 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:14.085373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:14.085826 (+ 453us) server_negotiation.cc:407] Connection header received
0620 23:22:14.086002 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:14.086007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:14.086091 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:14.086195 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:14.087686 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.088898 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.090885 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.091568 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.091694 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:14.097358 (+ 5664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:14.097365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:14.097370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:14.097409 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:14.100834 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:14.101634 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:14.101639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:14.101642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:14.101779 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:14.102180 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:14.102187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:14.102193 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:14.102539 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:14.102670 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:14.103253 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 23:22:14.103499 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:22:14.104568 6561 tablet_copy_service.cc:140] P c302f20b4e83403f9fe29b81eb2f7396: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer df8d3de65ed749678f8a3fb4c3d20588 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:53212)
I20260620 23:22:14.104740 6561 tablet_copy_service.cc:161] P c302f20b4e83403f9fe29b81eb2f7396: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer df8d3de65ed749678f8a3fb4c3d20588 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:53212: session id = df8d3de65ed749678f8a3fb4c3d20588-00000000000000000000000000000000
I20260620 23:22:14.108946 6561 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:22:14.110817 7102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:22:14.117377 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40977
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43331
--webserver_port=43503
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:43331,127.0.97.61:45803
--tserver_master_addrs=127.0.97.62:43331,127.0.97.61:45803,127.0.97.60:37761 with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:22:14.117784 7102 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: tablet copy: Starting download of 0 data blocks...
I20260620 23:22:14.118059 7102 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: tablet copy: Starting download of 1 WAL segments...
I20260620 23:22:14.120661 7102 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:22:14.124015 6561 tablet_copy_service.cc:342] P c302f20b4e83403f9fe29b81eb2f7396: Request end of tablet copy session df8d3de65ed749678f8a3fb4c3d20588-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:53212
I20260620 23:22:14.124157 6561 tablet_copy_service.cc:433] P c302f20b4e83403f9fe29b81eb2f7396: ending tablet copy session df8d3de65ed749678f8a3fb4c3d20588-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer df8d3de65ed749678f8a3fb4c3d20588
I20260620 23:22:14.136754 7102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:14.139770 7302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:14.145462 7303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:14.152834 7102 server_base.cc:1034] running on GCE node
W20260620 23:22:14.153210 7305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:14.153756 7102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:14.154343 7102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:14.155560 7102 hybrid_clock.cc:648] HybridClock initialized: now 1781997734155559 us; error 80 us; skew 500 ppm
I20260620 23:22:14.155807 7102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:14 dist-test-slave-5kzz krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997734, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:14.157935 7102 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 23:22:14.159520 7102 webserver.cc:460] Webserver started at http://127.0.97.60:34887/ using document root <none> and password file <none>
I20260620 23:22:14.159852 7102 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:14.159972 7102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:14.162784 7102 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:22:14.164656 7312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:14.165062 7102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:22:14.165215 7102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/wal
uuid: "df8d3de65ed749678f8a3fb4c3d20588"
format_stamp: "Formatted at 2026-06-20 23:22:13 on dist-test-slave-5kzz"
server_key: "4b11313a853fc05d58e15ca40cd45d7d"
server_key_iv: "a647b9df5aa79345a41177f5838762ff"
server_key_version: "encryptionkey@0"
I20260620 23:22:14.165467 7102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:14.232177 7102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:14.236403 7102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:14.266041 7102 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37761
I20260620 23:22:14.266235 7363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37761 every 8 connection(s)
I20260620 23:22:14.267215 7102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
I20260620 23:22:14.272758 7364 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:22:14.276854 7364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: Bootstrap starting.
I20260620 23:22:14.307909 7364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:22:14.308334 7364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: Bootstrap complete.
I20260620 23:22:14.309139 7364 raft_consensus.cc:348] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } }
I20260620 23:22:14.309367 7364 raft_consensus.cc:729] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: df8d3de65ed749678f8a3fb4c3d20588, State: Initialized, Role: LEARNER
I20260620 23:22:14.309638 7364 consensus_queue.cc:260] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } }
I20260620 23:22:14.310343 7367 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } } }
I20260620 23:22:14.310776 7367 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:22:14.310650 7364 sys_catalog.cc:564] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:14.318373 7378 catalog_manager.cc:1259] Loaded cluster ID: a322d1c8cca648a09cabb7b34ccf8e82
I20260620 23:22:14.318475 7378 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: loading cluster ID for follower catalog manager: success
I20260620 23:22:14.320751 7378 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: acquiring CA information for follower catalog manager: success
I20260620 23:22:14.322449 7378 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 23:22:14.526531 7301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:14.526863 7301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:14.527002 7301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:14.538067 7301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:14.538270 7301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:14.538467 7301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:22:14.545730 7383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:14.527797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.0.97.60:37761)
0620 23:22:14.539895 (+ 12098us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:14.539899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:14.539919 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:22:14.539977 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:14.539980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:14.540046 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:14.540147 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:14.540589 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.541550 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.544090 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.544901 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.545001 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:14.545133 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:14.545421 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 23:22:14.545512 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11989,"thread_start_us":11814,"threads_started":1}
I20260620 23:22:14.545825 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:14.527669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37761 (local address 127.0.0.1:45802)
0620 23:22:14.539371 (+ 11702us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:14.539390 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:14.539527 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:14.540189 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:14.540194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:14.540225 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:14.540415 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:14.540426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.542123 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.542128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:14.543951 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:14.543966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.545046 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.545053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:14.545271 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:14.545308 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:22:14.545398 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:22:14.545527 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11587,"thread_start_us":11503,"threads_started":1}
I20260620 23:22:14.546634 7339 raft_consensus.cc:1203] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 23:22:14.547250 7367 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c302f20b4e83403f9fe29b81eb2f7396. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } } }
I20260620 23:22:14.547544 7367 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:22:14.550822 7301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43503
--tserver_master_addrs=127.0.97.62:43331,127.0.97.61:45803,127.0.97.60:37761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:14.552011 7301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:14.553284 7301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:14.565786 7385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:14.565745 7386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:14.565765 7388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:14.566553 7301 server_base.cc:1034] running on GCE node
I20260620 23:22:14.567224 7301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:14.568064 7301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:14.569254 7301 hybrid_clock.cc:648] HybridClock initialized: now 1781997734569204 us; error 45 us; skew 500 ppm
I20260620 23:22:14.569553 7301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:14 dist-test-slave-5kzz krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997734, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:14.572502 7301 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 23:22:14.573725 7301 webserver.cc:460] Webserver started at http://127.0.97.1:43503/ using document root <none> and password file <none>
I20260620 23:22:14.574302 7301 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:14.574404 7301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:14.579066 7301 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:22:14.581694 7395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:14.582928 7301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:22:14.583067 7301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "86483dafc79b4aa289bdafbcb739d07e"
format_stamp: "Formatted at 2026-06-20 23:22:11 on dist-test-slave-5kzz"
server_key: "4e9617542434907546f3ba15ba694d5d"
server_key_iv: "a92a9d704316697237c9972e18bc95cc"
server_key_version: "encryptionkey@0"
I20260620 23:22:14.583880 7301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:14.636565 7301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:14.640733 7301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:14.640980 7301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:14.641855 7301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:14.643505 7301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:14.643612 7301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:14.643724 7301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:14.643812 7301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:14.676488 7301 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40977
I20260620 23:22:14.676563 7507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40977 every 8 connection(s)
I20260620 23:22:14.677850 7301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:22:14.682215 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 7301
I20260620 23:22:14.682576 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 6736
Jun 20 23:22:14 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997734, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 23:22:14 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997734, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 23:22:14.708727 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:14.680431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40219 (local address 127.0.97.62:43331)
0620 23:22:14.681069 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:14.681075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:14.682083 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 23:22:14.688969 (+ 6886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:14.688977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:14.689078 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:14.689171 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:14.690363 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.691770 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.694607 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.695439 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.695547 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:14.700400 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:14.700410 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:14.700419 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:14.700468 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:14.704211 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:14.705189 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:14.705196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:14.705201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:14.705349 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:14.706360 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:14.706366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:14.706369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:14.706730 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:14.706839 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:14.708102 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0620 23:22:14.708349 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":423,"threads_started":1}
Jun 20 23:22:14 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 23:22:14.711612 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:14.681066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.1:40219)
0620 23:22:14.681943 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:14.681987 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:14.685402 (+ 3415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:14.689420 (+ 4018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:14.689427 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:14.689949 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:14.690193 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:14.690204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.692001 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.692009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:14.694344 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:14.694360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.695857 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.695867 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:14.696151 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:14.696974 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:14.697005 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:14.700107 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:14.704381 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:14.704391 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:14.704410 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:14.705064 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:14.705493 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:14.705497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:14.705501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:14.706226 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:14.706955 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:14.706962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:14.707654 (+ 692us) client_negotiation.cc:769] Sending connection context
0620 23:22:14.709235 (+ 1581us) client_negotiation.cc:240] Negotiation successful
0620 23:22:14.709715 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"spinlock_wait_cycles":9984,"thread_start_us":420,"threads_started":1}
I20260620 23:22:14.711985 7383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:14.680532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55907 (local address 127.0.97.60:37761)
0620 23:22:14.680725 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:14.680731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:14.682089 (+ 1358us) server_negotiation.cc:407] Connection header received
0620 23:22:14.685873 (+ 3784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:14.685881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:14.685989 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:14.686071 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:14.694595 (+ 8524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.696084 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.698737 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.699604 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.699717 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:14.705192 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:14.705202 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:14.705209 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:14.705260 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:14.708864 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:14.709731 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:14.709739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:14.709743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:14.709890 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:14.710321 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:14.710338 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:14.710342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:14.710689 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:14.710883 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:14.711429 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 23:22:14.711645 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:22:14.713259 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:14.681066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37761 (local address 127.0.97.1:55907)
0620 23:22:14.681945 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:14.681989 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:14.685694 (+ 3705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:14.686649 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:14.686660 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:14.687766 (+ 1106us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:14.688720 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:14.688746 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.696318 (+ 7572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.696338 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:14.698504 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:14.698540 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.700131 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.700140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:14.700821 (+ 681us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:14.701533 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:14.701563 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:14.704950 (+ 3387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:14.709028 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:14.709033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:14.709038 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:14.709587 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:14.710035 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:14.710040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:14.710043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:14.710180 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:14.710969 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:14.710974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:14.711126 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 23:22:14.712453 (+ 1327us) client_negotiation.cc:240] Negotiation successful
0620 23:22:14.712754 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":24,"spinlock_wait_cycles":57984,"thread_start_us":1080,"threads_started":1}
I20260620 23:22:14.713332 7509 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43331
I20260620 23:22:14.713794 7509 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:14.714772 7509 heartbeater.cc:511] Master 127.0.97.62:43331 requested a full tablet report, sending...
I20260620 23:22:14.715224 7508 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37761
I20260620 23:22:14.715350 7508 heartbeater.cc:464] Registering TS with master...
Jun 20 23:22:14 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997734, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 23:22:14.715787 7508 heartbeater.cc:511] Master 127.0.97.60:37761 requested a full tablet report, sending...
I20260620 23:22:14.716719 6540 ts_manager.cc:194] Re-registered known tserver with Master: 86483dafc79b4aa289bdafbcb739d07e (127.0.97.1:40977)
I20260620 23:22:14.717466 7329 ts_manager.cc:194] Registered new tserver with Master: 86483dafc79b4aa289bdafbcb739d07e (127.0.97.1:40977)
I20260620 23:22:14.718030 6540 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40219
I20260620 23:22:14.720180 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:14.685657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.0.97.2:42345)
0620 23:22:14.693938 (+ 8281us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:14.693946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:14.693973 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:22:14.694041 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:14.694048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:14.694275 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:14.697663 (+ 3388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:14.697885 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.699423 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.704104 (+ 4681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.704940 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.705057 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:14.711735 (+ 6678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:14.711747 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:14.711802 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:14.711854 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:14.715945 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:14.716698 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:14.716706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:14.716708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:14.716850 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:14.717307 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:14.717312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:14.717315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:14.717695 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:14.718390 (+ 695us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:14.719461 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0620 23:22:14.719696 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8112,"thread_start_us":7990,"threads_started":1}
I20260620 23:22:14.721424 6859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:14.729002 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:14.682655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45803 (local address 127.0.97.1:34061)
0620 23:22:14.706294 (+ 23639us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:14.706313 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:14.706405 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:14.707493 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:14.707497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:14.708079 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:14.708357 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:14.708367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.709759 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.709763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:14.711730 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:14.711747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.712861 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.712868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:14.713041 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:14.713571 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:14.713592 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:14.716230 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:14.726975 (+ 10745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:14.726980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:14.726984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:14.727315 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:14.727692 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:14.727695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:14.727697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:14.727815 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:14.728380 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:14.728382 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:14.728473 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 23:22:14.728563 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:22:14.728754 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23500,"thread_start_us":24598,"threads_started":1}
I20260620 23:22:14.729306 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:14.682923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34061 (local address 127.0.97.61:45803)
0620 23:22:14.706823 (+ 23900us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:14.706829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:14.706854 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:22:14.706930 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:14.706935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:14.707017 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:14.707122 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:14.708557 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.709591 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.711922 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:14.712500 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:14.712608 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:14.718999 (+ 6391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:14.719010 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:14.719016 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:14.719056 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:14.726811 (+ 7755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:14.727452 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:14.727457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:14.727461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:14.727589 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:14.727923 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:14.727929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:14.727933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:14.728278 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:14.728413 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:14.728696 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 23:22:14.728909 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23766,"thread_start_us":23657,"threads_started":1}
I20260620 23:22:14.730188 7510 heartbeater.cc:347] Connected to a master server at 127.0.97.61:45803
I20260620 23:22:14.730326 7510 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:14.730728 7510 heartbeater.cc:511] Master 127.0.97.61:45803 requested a full tablet report, sending...
I20260620 23:22:14.732429 7138 ts_manager.cc:194] Registered new tserver with Master: 86483dafc79b4aa289bdafbcb739d07e (127.0.97.1:40977)
I20260620 23:22:14.802184 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 6736
I20260620 23:22:14.818620 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42345
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43331
--webserver_port=36535
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:43331,127.0.97.61:45803
--tserver_master_addrs=127.0.97.62:43331,127.0.97.61:45803,127.0.97.60:37761 with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:22:14.937490 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: true } } }
I20260620 23:22:14.937906 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:14.939288 7521 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396: attempting to promote NON_VOTER df8d3de65ed749678f8a3fb4c3d20588 to VOTER
I20260620 23:22:14.939829 7521 consensus_queue.cc:237] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } }
I20260620 23:22:14.941504 7148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 FOLLOWER]: Refusing update from remote peer c302f20b4e83403f9fe29b81eb2f7396: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 23:22:14.941744 7339 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 LEARNER]: Refusing update from remote peer c302f20b4e83403f9fe29b81eb2f7396: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 23:22:14.942158 7522 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 23:22:14.942555 7287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 23:22:14.945608 7148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, df8d3de65ed749678f8a3fb4c3d20588 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } } }
I20260620 23:22:14.947094 7339 raft_consensus.cc:2932] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, df8d3de65ed749678f8a3fb4c3d20588 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } } }
I20260620 23:22:14.950093 7367 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } } }
I20260620 23:22:14.950317 7367 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:22:14.944876 7522 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, df8d3de65ed749678f8a3fb4c3d20588 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } } }
I20260620 23:22:14.953229 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } } }
I20260620 23:22:14.953524 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:14.953408 7522 sys_catalog.cc:455] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } } }
I20260620 23:22:14.953641 7522 sys_catalog.cc:458] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:14.953696 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } } }
I20260620 23:22:14.953994 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:14.959056 7525 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c302f20b4e83403f9fe29b81eb2f7396" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } } }
I20260620 23:22:14.959492 7525 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:22:15.142602 7520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:15.142903 7520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:15.143055 7520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:15.149927 7520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:15.150064 7520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:15.150218 7520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:22:15.159339 7520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36535
--tserver_master_addrs=127.0.97.62:43331,127.0.97.61:45803,127.0.97.60:37761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:15.160537 7520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:15.161773 7520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:15.174602 7534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:15.174661 7532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:15.174872 7531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:15.175354 7520 server_base.cc:1034] running on GCE node
I20260620 23:22:15.176021 7520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:15.176816 7520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:15.178004 7520 hybrid_clock.cc:648] HybridClock initialized: now 1781997735177942 us; error 52 us; skew 500 ppm
I20260620 23:22:15.178296 7520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:15 dist-test-slave-5kzz krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997735, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:15.181140 7520 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 23:22:15.182361 7520 webserver.cc:460] Webserver started at http://127.0.97.2:36535/ using document root <none> and password file <none>
I20260620 23:22:15.183040 7520 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:15.183182 7520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:15.188058 7520 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:22:15.190361 7541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:15.191552 7520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 23:22:15.191711 7520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "218b5570d52c4d6dab9617e51a26792d"
format_stamp: "Formatted at 2026-06-20 23:22:12 on dist-test-slave-5kzz"
server_key: "58a25efba5636d28c41b662f9c229c45"
server_key_iv: "00ea7ab31673e6adcc6685ec9224e027"
server_key_version: "encryptionkey@0"
I20260620 23:22:15.192593 7520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:15.207175 7520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:15.210825 7520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:15.211133 7520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:15.212074 7520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:15.213322 7520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:15.213454 7520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:15.213567 7520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:15.213653 7520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:15.245426 7520 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42345
I20260620 23:22:15.245507 7653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42345 every 8 connection(s)
I20260620 23:22:15.246932 7520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:22:15.256639 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 7520
I20260620 23:22:15.256994 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 6869
Jun 20 23:22:15 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997735, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 23:22:15 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997735, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 23:22:15 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 23:22:15.285655 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.258143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.0.97.3:38711)
0620 23:22:15.263894 (+ 5751us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.263902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.263926 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:22:15.263982 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.263988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.264199 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.266281 (+ 2082us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:15.266530 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.268425 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.270099 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.270817 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.277163 (+ 6346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.277467 (+ 304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:15.277480 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:15.277495 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:15.277532 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:15.281584 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.282440 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.282451 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.282455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.282600 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.283072 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.283079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.283085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.283490 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:15.284840 (+ 1350us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.284978 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.285233 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5608,"thread_start_us":5462,"threads_started":1}
I20260620 23:22:15.287329 6992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:15.287629 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.249528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38383 (local address 127.0.97.62:43331)
0620 23:22:15.258128 (+ 8600us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.258135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.260228 (+ 2093us) server_negotiation.cc:407] Connection header received
0620 23:22:15.262061 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.262066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.262163 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.262261 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:15.264371 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.265505 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.268516 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.269153 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.269246 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.275295 (+ 6049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:15.275303 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:15.275310 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:15.275352 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:15.279074 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.280914 (+ 1840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.280920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.280926 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.281067 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.285899 (+ 4832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.285905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.285908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.286218 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:15.286343 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.286923 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.287221 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8457,"thread_start_us":8342,"threads_started":1}
I20260620 23:22:15.288790 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.249612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47365 (local address 127.0.97.61:45803)
0620 23:22:15.261519 (+ 11907us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.261529 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.263864 (+ 2335us) server_negotiation.cc:407] Connection header received
0620 23:22:15.263947 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.263955 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.264049 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.264176 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:15.265522 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.266545 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.268752 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.269331 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.269442 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.280103 (+ 10661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:15.280109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:15.280113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:15.280168 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:15.283694 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.284470 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.284475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.284480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.284616 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.285382 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.285387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.285390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.285725 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:15.285870 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.288194 (+ 2324us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.288476 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11752,"thread_start_us":13205,"threads_started":1}
I20260620 23:22:15.289700 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.250277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45803 (local address 127.0.97.2:47365)
0620 23:22:15.263300 (+ 13023us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:15.263315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:15.263437 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:15.264378 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:15.264385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:15.265064 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:15.265323 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.265333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.266748 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.266752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.268579 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.268593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.269672 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.269678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.269856 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.270785 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:15.270806 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:15.275069 (+ 4263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:15.283837 (+ 8768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:15.283842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:15.283845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:15.284329 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:15.285115 (+ 786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:15.285119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:15.285123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:15.285243 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:15.285936 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:15.285946 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:15.286339 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 23:22:15.286728 (+ 389us) client_negotiation.cc:240] Negotiation successful
0620 23:22:15.287231 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12871,"mutex_wait_us":64,"spinlock_wait_cycles":56960,"thread_start_us":12546,"threads_started":1}
I20260620 23:22:15.290578 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.250277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.2:38383)
0620 23:22:15.260079 (+ 9802us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:15.260121 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:15.261758 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:15.262646 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:15.262656 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:15.263448 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:15.264202 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.264223 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.265699 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.265705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.268349 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.268363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.269531 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.269538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.269797 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.270642 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:15.270670 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:15.273901 (+ 3231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:15.279257 (+ 5356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:15.279266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:15.279283 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:15.279921 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:15.284773 (+ 4852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:15.284779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:15.284783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:15.285014 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:15.286484 (+ 1470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:15.286488 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:15.286654 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 23:22:15.288291 (+ 1637us) client_negotiation.cc:240] Negotiation successful
0620 23:22:15.288558 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9553,"mutex_wait_us":29,"spinlock_wait_cycles":3840,"thread_start_us":11656,"threads_started":1}
I20260620 23:22:15.291585 7655 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43331
I20260620 23:22:15.291884 7655 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:15.292730 7655 heartbeater.cc:511] Master 127.0.97.62:43331 requested a full tablet report, sending...
I20260620 23:22:15.294276 6540 ts_manager.cc:194] Re-registered known tserver with Master: 218b5570d52c4d6dab9617e51a26792d (127.0.97.2:42345)
I20260620 23:22:15.295720 6540 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38383
Jun 20 23:22:15 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997735, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 23:22:15.298592 7656 heartbeater.cc:347] Connected to a master server at 127.0.97.61:45803
I20260620 23:22:15.298754 7656 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:15.299188 7656 heartbeater.cc:511] Master 127.0.97.61:45803 requested a full tablet report, sending...
I20260620 23:22:15.300522 7138 ts_manager.cc:194] Registered new tserver with Master: 218b5570d52c4d6dab9617e51a26792d (127.0.97.2:42345)
I20260620 23:22:15.304515 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.263663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42565 (local address 127.0.97.60:37761)
0620 23:22:15.271476 (+ 7813us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.271482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.286765 (+ 15283us) server_negotiation.cc:407] Connection header received
0620 23:22:15.287032 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.287037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.287116 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.287198 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:15.289136 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.291035 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.292938 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.293567 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.293668 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.297742 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:15.297749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:15.297754 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:15.297789 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:15.301686 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.302499 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.302506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.302533 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.302759 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.303225 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.303230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.303233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.303500 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:15.303653 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.303987 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.304176 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7694,"thread_start_us":7576,"threads_started":1}
I20260620 23:22:15.304566 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.271610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37761 (local address 127.0.97.2:42565)
0620 23:22:15.286637 (+ 15027us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:15.286657 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:15.286867 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:15.287439 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:15.287447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:15.288519 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:15.288884 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.288898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.291268 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.291272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.292783 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.292798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.293950 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.293956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.294106 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.294870 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:15.294891 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:15.297518 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:15.301825 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:15.301832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:15.301836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:15.302374 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:15.302878 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:15.302886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:15.302892 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:15.303113 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:15.303626 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:15.303630 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:15.303762 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 23:22:15.303900 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 23:22:15.304171 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14907,"thread_start_us":14640,"threads_started":1}
I20260620 23:22:15.305217 7654 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37761
I20260620 23:22:15.305347 7654 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:15.305625 7654 heartbeater.cc:511] Master 127.0.97.60:37761 requested a full tablet report, sending...
I20260620 23:22:15.306504 7329 ts_manager.cc:194] Registered new tserver with Master: 218b5570d52c4d6dab9617e51a26792d (127.0.97.2:42345)
I20260620 23:22:15.379771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 6869
I20260620 23:22:15.395056 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38711
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43331
--webserver_port=46573
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:43331,127.0.97.61:45803
--tserver_master_addrs=127.0.97.62:43331,127.0.97.61:45803,127.0.97.60:37761 with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:15.709164 7667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:15.709408 7667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:15.709499 7667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:15.716759 7667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:15.716874 7667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:15.717078 7667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:22:15.720767 7509 heartbeater.cc:503] Master 127.0.97.62:43331 was elected leader, sending a full tablet report...
I20260620 23:22:15.726576 7667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46573
--tserver_master_addrs=127.0.97.62:43331,127.0.97.61:45803,127.0.97.60:37761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:15.727759 7667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:15.728884 7667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:15.741734 7672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:15.741721 7673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:15.741750 7675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:15.742653 7667 server_base.cc:1034] running on GCE node
I20260620 23:22:15.743355 7667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:15.744318 7667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:15.745510 7667 hybrid_clock.cc:648] HybridClock initialized: now 1781997735745460 us; error 43 us; skew 500 ppm
I20260620 23:22:15.745798 7667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:15 dist-test-slave-5kzz krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997735, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:15.748729 7667 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 23:22:15.750118 7667 webserver.cc:460] Webserver started at http://127.0.97.3:46573/ using document root <none> and password file <none>
I20260620 23:22:15.750805 7667 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:15.750958 7667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:15.756027 7667 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:22:15.758477 7682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:15.759642 7667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 23:22:15.759816 7667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "d71f0e04eac14825a32248eaa74d2c86"
format_stamp: "Formatted at 2026-06-20 23:22:12 on dist-test-slave-5kzz"
server_key: "a14baeb5b77849f9369189e846c93e91"
server_key_iv: "15fd15a0cded3c0c32e864bbfa845ee8"
server_key_version: "encryptionkey@0"
I20260620 23:22:15.760692 7667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:15.776374 7667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:15.780278 7667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:15.780546 7667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:15.781414 7667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:15.782588 7667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:15.782720 7667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:15.782842 7667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:15.782927 7667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:15.814047 7667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38711
I20260620 23:22:15.814126 7794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38711 every 8 connection(s)
I20260620 23:22:15.815456 7667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:22:15.822886 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 7667
Jun 20 23:22:15 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997735, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 23:22:15 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997735, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 23:22:15 dist-test-slave-5kzz krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997735, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 23:22:15.850515 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.818150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60221 (local address 127.0.97.60:37761)
0620 23:22:15.819846 (+ 1696us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.819850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.820428 (+ 578us) server_negotiation.cc:407] Connection header received
0620 23:22:15.822200 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.822205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.822294 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.822420 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:15.824835 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.825947 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.828323 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.828938 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.829033 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.841166 (+ 12133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:15.841175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:15.841182 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:15.841229 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:15.845498 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.847181 (+ 1683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.847187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.847192 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.847358 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.848243 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.848248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.848252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.848600 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:15.848727 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.849886 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.850116 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1584,"thread_start_us":1512,"threads_started":1}
I20260620 23:22:15.853135 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.824076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45803 (local address 127.0.97.3:57993)
0620 23:22:15.824603 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:15.824619 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:15.824714 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:15.830542 (+ 5828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:15.830547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:15.831802 (+ 1255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:15.832173 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.832187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.835456 (+ 3269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.835464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.837432 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.837450 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.838783 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.838792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.839049 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.839826 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:15.839856 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:15.843162 (+ 3306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:15.848245 (+ 5083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:15.848251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:15.848257 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:15.848747 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:15.849156 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:15.849163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:15.849169 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:15.849314 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:15.849981 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:15.849988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:15.850125 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 23:22:15.850280 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 23:22:15.852711 (+ 2431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":280,"threads_started":1}
I20260620 23:22:15.852722 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.818660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37761 (local address 127.0.97.3:60221)
0620 23:22:15.820274 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:15.820311 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:15.821932 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:15.823324 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:15.823334 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:15.823970 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:15.824694 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.824715 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.826116 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.826121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.828164 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.828190 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.829326 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.829333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.829598 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.830995 (+ 1397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:15.831023 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:15.839795 (+ 8772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:15.845696 (+ 5901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:15.845708 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:15.845729 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:15.846299 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:15.847538 (+ 1239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:15.847543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:15.847547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:15.847760 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:15.849008 (+ 1248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:15.849016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:15.849452 (+ 436us) client_negotiation.cc:769] Sending connection context
0620 23:22:15.850780 (+ 1328us) client_negotiation.cc:240] Negotiation successful
0620 23:22:15.851112 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1364,"thread_start_us":3534,"threads_started":1}
I20260620 23:22:15.854321 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.822953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57993 (local address 127.0.97.61:45803)
0620 23:22:15.830203 (+ 7250us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.830208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.830226 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:22:15.830291 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.830295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.830388 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.830751 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:15.832346 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.835239 (+ 2893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.837666 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.838531 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.843388 (+ 4857us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.843640 (+ 252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:15.843649 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:15.843656 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:15.843692 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:15.848079 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.848892 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.848896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.848900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.849044 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.849467 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.849472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.849476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.849843 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:15.852806 (+ 2963us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.852976 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.853178 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7125,"thread_start_us":7480,"threads_started":1}
I20260620 23:22:15.854889 7795 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37761
I20260620 23:22:15.855242 7795 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:15.855581 7797 heartbeater.cc:347] Connected to a master server at 127.0.97.61:45803
I20260620 23:22:15.855707 7797 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:15.856225 7795 heartbeater.cc:511] Master 127.0.97.60:37761 requested a full tablet report, sending...
I20260620 23:22:15.856200 7797 heartbeater.cc:511] Master 127.0.97.61:45803 requested a full tablet report, sending...
I20260620 23:22:15.857929 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.818661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.3:44867)
0620 23:22:15.822683 (+ 4022us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:15.822717 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:15.822884 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:15.824884 (+ 2000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:15.824890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:15.825378 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:15.825599 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.825609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.827065 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.827070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.828508 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.828519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.829567 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.829572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.829706 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.830514 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:15.830540 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:15.844790 (+ 14250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:15.850002 (+ 5212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:15.850006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:15.850009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:15.850464 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:15.852515 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:15.852519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:15.852523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:15.852727 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:15.856802 (+ 4075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:15.856810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:15.857017 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 23:22:15.857221 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 23:22:15.857494 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3824,"mutex_wait_us":335,"thread_start_us":907,"threads_started":1}
I20260620 23:22:15.858121 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.818372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44867 (local address 127.0.97.62:43331)
0620 23:22:15.819581 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.819586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.822786 (+ 3200us) server_negotiation.cc:407] Connection header received
0620 23:22:15.823088 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.823093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.823171 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.823270 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:15.825772 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.826866 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.828644 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.829251 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.829333 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.846146 (+ 16813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:15.846154 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:15.846159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:15.846195 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:15.849794 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.850946 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.850951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.850954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.851101 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.852922 (+ 1821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.852929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.852932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.853511 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:15.854066 (+ 555us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.857368 (+ 3302us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.857635 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100,"thread_start_us":1034,"threads_started":1}
I20260620 23:22:15.858234 7329 ts_manager.cc:194] Registered new tserver with Master: d71f0e04eac14825a32248eaa74d2c86 (127.0.97.3:38711)
I20260620 23:22:15.859335 7796 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43331
I20260620 23:22:15.859460 7796 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:15.859575 7138 ts_manager.cc:194] Registered new tserver with Master: d71f0e04eac14825a32248eaa74d2c86 (127.0.97.3:38711)
I20260620 23:22:15.859889 7796 heartbeater.cc:511] Master 127.0.97.62:43331 requested a full tablet report, sending...
I20260620 23:22:15.860882 6540 ts_manager.cc:194] Re-registered known tserver with Master: d71f0e04eac14825a32248eaa74d2c86 (127.0.97.3:38711)
I20260620 23:22:15.861759 6540 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44867
I20260620 23:22:15.869717 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.833149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.0.97.62:43331)
0620 23:22:15.848428 (+ 15279us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.848435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.849031 (+ 596us) server_negotiation.cc:407] Connection header received
0620 23:22:15.849265 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.849270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.849335 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.849411 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:15.851801 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.852854 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.855170 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.855917 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.856005 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.862917 (+ 6912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:15.862926 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:15.862930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:15.862965 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:15.866618 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.867596 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.867603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.867607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.867778 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.868227 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.868233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.868236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.868483 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:15.868752 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.869081 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.869289 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15172,"thread_start_us":15286,"threads_started":1}
I20260620 23:22:15.880445 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.863799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.0.97.60:37761)
0620 23:22:15.864000 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.864007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.864984 (+ 977us) server_negotiation.cc:407] Connection header received
0620 23:22:15.865200 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.865206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.865278 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.865369 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:15.867829 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.868904 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.870610 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.871229 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.871321 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.874652 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:15.874658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:15.874662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:15.874725 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:15.877493 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.878443 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.878452 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.878455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.878598 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.879025 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.879031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.879035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.879405 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:15.879633 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.879928 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.880124 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:22:15.881451 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.833383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.0.97.61:45803)
0620 23:22:15.855290 (+ 21907us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.855296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.863979 (+ 8683us) server_negotiation.cc:407] Connection header received
0620 23:22:15.864038 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.864045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.864131 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.864229 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:15.866583 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.867632 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.869434 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.870018 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.870103 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.873682 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:15.873691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:15.873696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:15.873736 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:15.877684 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.878509 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.878515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.878518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.878644 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:15.879709 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:15.879715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:15.879718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:15.880079 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:15.880334 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.880923 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.881133 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21765,"thread_start_us":30150,"threads_started":1}
I20260620 23:22:15.873808 6540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:22:15.883916 6540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:22:15.918392 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.902134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.0.97.1:40977)
0620 23:22:15.907142 (+ 5008us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.907150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.907570 (+ 420us) server_negotiation.cc:407] Connection header received
0620 23:22:15.907765 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.907773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.907979 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.908612 (+ 633us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:15.909350 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.910351 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.914872 (+ 4521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.916613 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.916739 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.917043 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.917816 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.918019 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4851,"thread_start_us":4725,"threads_started":1}
I20260620 23:22:15.918535 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.907841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.0.97.2:42345)
0620 23:22:15.908832 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.908840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.911367 (+ 2527us) server_negotiation.cc:407] Connection header received
0620 23:22:15.911465 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.911472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.911776 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.912000 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:15.912929 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.914930 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.916523 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.917520 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.917657 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.917851 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.918089 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.918246 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":697,"threads_started":1}
I20260620 23:22:15.918653 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.907578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42345 (local address 127.0.0.1:35118)
0620 23:22:15.911093 (+ 3515us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:15.911111 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:15.911237 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:15.912242 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:15.912250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:15.912282 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:15.912693 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.912706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.915088 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.915092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.916398 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.916408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.917713 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.917720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.917906 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.917942 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:22:15.918055 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:22:15.918190 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3392,"thread_start_us":3247,"threads_started":1}
I20260620 23:22:15.920291 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.901987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40977 (local address 127.0.0.1:50736)
0620 23:22:15.907456 (+ 5469us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:15.907470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:15.907613 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:15.908822 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:15.908827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:15.908861 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:15.909169 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.909178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.910512 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.910520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.914664 (+ 4144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.914697 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.917298 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.917307 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.917550 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.917601 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:22:15.919491 (+ 1890us) client_negotiation.cc:240] Negotiation successful
0620 23:22:15.919640 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5379,"thread_start_us":5275,"threads_started":1}
I20260620 23:22:15.923527 7589 tablet_service.cc:1467] Processing CreateTablet for tablet 0d82a877d8ac46b3a09dd4f4c07bcde9 (DEFAULT_TABLE table=test-table [id=85ce1e0d97fc4c159efe4211a09dd640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:15.923830 7443 tablet_service.cc:1467] Processing CreateTablet for tablet 0d82a877d8ac46b3a09dd4f4c07bcde9 (DEFAULT_TABLE table=test-table [id=85ce1e0d97fc4c159efe4211a09dd640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:15.924839 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d82a877d8ac46b3a09dd4f4c07bcde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:15.925267 7443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d82a877d8ac46b3a09dd4f4c07bcde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:15.926788 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.911106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38711 (local address 127.0.0.1:57152)
0620 23:22:15.918158 (+ 7052us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:15.918174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:15.918300 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:15.919424 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:15.919427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:15.919488 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:15.919899 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.919909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.922105 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.922110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.923971 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.923985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.926034 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.926039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.926205 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.926239 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:22:15.926335 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:22:15.926443 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6938,"thread_start_us":8552,"threads_started":1}
I20260620 23:22:15.926865 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.911486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.0.97.3:38711)
0620 23:22:15.915199 (+ 3713us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.915208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.918422 (+ 3214us) server_negotiation.cc:407] Connection header received
0620 23:22:15.918514 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.918521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.918803 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.919431 (+ 628us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:15.920190 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.921958 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.924120 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.925694 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.925937 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.926126 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.926385 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.926533 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3556,"thread_start_us":3389,"threads_started":1}
I20260620 23:22:15.930315 7730 tablet_service.cc:1467] Processing CreateTablet for tablet 0d82a877d8ac46b3a09dd4f4c07bcde9 (DEFAULT_TABLE table=test-table [id=85ce1e0d97fc4c159efe4211a09dd640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:15.931588 7730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d82a877d8ac46b3a09dd4f4c07bcde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:15.937206 7828 tablet_bootstrap.cc:492] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e: Bootstrap starting.
I20260620 23:22:15.939445 7829 tablet_bootstrap.cc:492] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d: Bootstrap starting.
I20260620 23:22:15.941573 7828 tablet_bootstrap.cc:654] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:15.942515 7829 tablet_bootstrap.cc:654] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:15.942756 7828 log.cc:824] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:15.943652 7829 log.cc:824] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:15.945765 7828 tablet_bootstrap.cc:492] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e: No bootstrap required, opened a new log
I20260620 23:22:15.946009 7828 ts_tablet_manager.cc:1397] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260620 23:22:15.946412 7829 tablet_bootstrap.cc:492] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d: No bootstrap required, opened a new log
I20260620 23:22:15.947140 7829 ts_tablet_manager.cc:1397] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260620 23:22:15.950942 7832 tablet_bootstrap.cc:492] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86: Bootstrap starting.
I20260620 23:22:15.955029 7832 tablet_bootstrap.cc:654] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:15.956108 7832 log.cc:824] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:15.957150 7829 raft_consensus.cc:348] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:15.957150 7828 raft_consensus.cc:348] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:15.957561 7829 raft_consensus.cc:374] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:15.957561 7828 raft_consensus.cc:374] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:15.957656 7828 raft_consensus.cc:729] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86483dafc79b4aa289bdafbcb739d07e, State: Initialized, Role: FOLLOWER
I20260620 23:22:15.957657 7829 raft_consensus.cc:729] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218b5570d52c4d6dab9617e51a26792d, State: Initialized, Role: FOLLOWER
I20260620 23:22:15.958268 7828 consensus_queue.cc:260] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:15.958262 7829 consensus_queue.cc:260] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:15.958868 7832 tablet_bootstrap.cc:492] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86: No bootstrap required, opened a new log
I20260620 23:22:15.959139 7832 ts_tablet_manager.cc:1397] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260620 23:22:15.959748 7655 heartbeater.cc:503] Master 127.0.97.62:43331 was elected leader, sending a full tablet report...
I20260620 23:22:15.960018 7829 ts_tablet_manager.cc:1428] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260620 23:22:15.961561 7828 ts_tablet_manager.cc:1428] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260620 23:22:15.972532 7835 raft_consensus.cc:482] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:15.972553 7832 raft_consensus.cc:348] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:15.972946 7832 raft_consensus.cc:374] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:15.972785 7835 raft_consensus.cc:504] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:15.973047 7832 raft_consensus.cc:729] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71f0e04eac14825a32248eaa74d2c86, State: Initialized, Role: FOLLOWER
I20260620 23:22:15.973583 7832 consensus_queue.cc:260] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:15.974597 7796 heartbeater.cc:503] Master 127.0.97.62:43331 was elected leader, sending a full tablet report...
I20260620 23:22:15.975322 7832 ts_tablet_manager.cc:1428] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86: Time spent starting tablet: real 0.016s user 0.006s sys 0.004s
I20260620 23:22:15.977564 7835 leader_election.cc:288] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218b5570d52c4d6dab9617e51a26792d (127.0.97.2:42345), d71f0e04eac14825a32248eaa74d2c86 (127.0.97.3:38711)
I20260620 23:22:15.984720 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.975068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35027 (local address 127.0.97.2:42345)
0620 23:22:15.975256 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.975262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.978204 (+ 2942us) server_negotiation.cc:407] Connection header received
0620 23:22:15.978519 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.978524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.978599 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.978660 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:15.980283 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.981047 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.982989 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.983856 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.983998 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.984092 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.984360 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.984475 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:22:15.984849 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.974726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42345 (local address 127.0.97.1:35027)
0620 23:22:15.978118 (+ 3392us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:15.978138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:15.978354 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:15.979773 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:15.979779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:15.979814 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:15.980121 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.980134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.981174 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.981179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.982857 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.982868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.984045 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.984051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.984194 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.984217 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:22:15.984314 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:22:15.984435 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3266,"mutex_wait_us":1,"thread_start_us":3077,"threads_started":1}
I20260620 23:22:15.985152 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.978360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57961 (local address 127.0.97.3:38711)
0620 23:22:15.978561 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:15.978570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:15.980094 (+ 1524us) server_negotiation.cc:407] Connection header received
0620 23:22:15.980321 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:15.980325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:15.980399 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:15.980455 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:15.980949 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.981852 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.983528 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.984278 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.984426 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.984524 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:15.984783 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 23:22:15.984921 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:22:15.985463 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:15.978347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38711 (local address 127.0.97.1:57961)
0620 23:22:15.980002 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:15.980018 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:15.980183 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:15.980584 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:15.980587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:15.980603 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:15.980820 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.980826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.981978 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.981986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.983391 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:15.983401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:15.984487 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:15.984493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:15.984634 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:15.984656 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:22:15.984761 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:22:15.985138 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1304,"mutex_wait_us":18,"thread_start_us":1183,"threads_started":1}
I20260620 23:22:15.985760 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d82a877d8ac46b3a09dd4f4c07bcde9" candidate_uuid: "86483dafc79b4aa289bdafbcb739d07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218b5570d52c4d6dab9617e51a26792d" is_pre_election: true
I20260620 23:22:15.986213 7609 raft_consensus.cc:2454] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86483dafc79b4aa289bdafbcb739d07e in term 0.
I20260620 23:22:15.986218 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d82a877d8ac46b3a09dd4f4c07bcde9" candidate_uuid: "86483dafc79b4aa289bdafbcb739d07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d71f0e04eac14825a32248eaa74d2c86" is_pre_election: true
I20260620 23:22:15.986637 7750 raft_consensus.cc:2454] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86483dafc79b4aa289bdafbcb739d07e in term 0.
I20260620 23:22:15.986948 7397 leader_election.cc:302] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218b5570d52c4d6dab9617e51a26792d, 86483dafc79b4aa289bdafbcb739d07e; no voters:
I20260620 23:22:15.987298 7835 raft_consensus.cc:2781] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:22:15.987452 7835 raft_consensus.cc:482] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:22:15.987560 7835 raft_consensus.cc:3037] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:15.989992 7835 raft_consensus.cc:504] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:15.990617 7835 leader_election.cc:288] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [CANDIDATE]: Term 1 election: Requested vote from peers 218b5570d52c4d6dab9617e51a26792d (127.0.97.2:42345), d71f0e04eac14825a32248eaa74d2c86 (127.0.97.3:38711)
I20260620 23:22:15.990933 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d82a877d8ac46b3a09dd4f4c07bcde9" candidate_uuid: "86483dafc79b4aa289bdafbcb739d07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218b5570d52c4d6dab9617e51a26792d"
I20260620 23:22:15.991132 7609 raft_consensus.cc:3037] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:15.991700 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d82a877d8ac46b3a09dd4f4c07bcde9" candidate_uuid: "86483dafc79b4aa289bdafbcb739d07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d71f0e04eac14825a32248eaa74d2c86"
I20260620 23:22:15.991904 7750 raft_consensus.cc:3037] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:15.993707 7609 raft_consensus.cc:2454] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86483dafc79b4aa289bdafbcb739d07e in term 1.
I20260620 23:22:15.994069 7397 leader_election.cc:302] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218b5570d52c4d6dab9617e51a26792d, 86483dafc79b4aa289bdafbcb739d07e; no voters:
I20260620 23:22:15.994264 7750 raft_consensus.cc:2454] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86483dafc79b4aa289bdafbcb739d07e in term 1.
I20260620 23:22:15.994297 7835 raft_consensus.cc:2781] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:15.994779 7835 raft_consensus.cc:686] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 1 LEADER]: Becoming Leader. State: Replica: 86483dafc79b4aa289bdafbcb739d07e, State: Running, Role: LEADER
I20260620 23:22:15.995368 7835 consensus_queue.cc:237] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:15.999910 6541 catalog_manager.cc:5495] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e reported cstate change: term changed from 0 to 1, leader changed from <none> to 86483dafc79b4aa289bdafbcb739d07e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "86483dafc79b4aa289bdafbcb739d07e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } health_report { overall_health: UNKNOWN } } }
W20260620 23:22:16.000373 7657 tablet.cc:2369] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:22:16.036128 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:16.031219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.0.97.1:40977)
0620 23:22:16.031426 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:16.031433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:16.031450 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:22:16.031661 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:16.031664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:16.031736 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:16.031877 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:22:16.032223 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:16.032868 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:16.034095 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:16.034474 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:16.034607 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:16.035101 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:22:16.035563 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:22:16.035741 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:16.035790 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 23:22:16.035867 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:22:16.043430 7750 raft_consensus.cc:1261] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86 [term 1 FOLLOWER]: Refusing update from remote peer 86483dafc79b4aa289bdafbcb739d07e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:16.043479 7609 raft_consensus.cc:1261] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d [term 1 FOLLOWER]: Refusing update from remote peer 86483dafc79b4aa289bdafbcb739d07e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:16.044353 7839 consensus_queue.cc:1035] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:16.044677 7835 consensus_queue.cc:1035] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [LEADER]: Connected to new peer: Peer: permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:16.082113 6540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:16.082396 6540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:16.088924 6540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c302f20b4e83403f9fe29b81eb2f7396: Sending DeleteTablet for 3 replicas of tablet 0d82a877d8ac46b3a09dd4f4c07bcde9
I20260620 23:22:16.090154 7589 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d82a877d8ac46b3a09dd4f4c07bcde9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:16 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35118
I20260620 23:22:16.090339 7730 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d82a877d8ac46b3a09dd4f4c07bcde9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:16 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57152
I20260620 23:22:16.091030 7852 tablet_replica.cc:330] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d: stopping tablet replica
I20260620 23:22:16.091408 7852 raft_consensus.cc:2229] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:16.091367 7443 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d82a877d8ac46b3a09dd4f4c07bcde9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:16 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50736
I20260620 23:22:16.091849 7852 raft_consensus.cc:2258] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:16.096251 7852 ts_tablet_manager.cc:1905] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:16.096805 7853 tablet_replica.cc:330] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86: stopping tablet replica
I20260620 23:22:16.097532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 6509
I20260620 23:22:16.098443 6571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:16.101366 7854 tablet_replica.cc:330] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e: stopping tablet replica
I20260620 23:22:16.101958 7854 raft_consensus.cc:2229] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:22:16.101126 7853 raft_consensus.cc:2229] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:16.103883 7853 raft_consensus.cc:2258] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:16.104522 7854 raft_consensus.cc:2258] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:16.105851 7852 ts_tablet_manager.cc:1918] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:16.106290 7852 log.cc:1196] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/0d82a877d8ac46b3a09dd4f4c07bcde9
I20260620 23:22:16.106863 7852 ts_tablet_manager.cc:1939] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 218b5570d52c4d6dab9617e51a26792d: Deleting consensus metadata
I20260620 23:22:16.109102 7853 ts_tablet_manager.cc:1905] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:16.112691 7854 ts_tablet_manager.cc:1905] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:16.122617 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:16.093576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.0.97.61:45803)
0620 23:22:16.093744 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:16.093749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:16.093767 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:22:16.094026 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:16.094032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:16.094101 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:16.094171 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:16.098734 (+ 4563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:16.099810 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:16.106065 (+ 6255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:16.106789 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:16.106892 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:16.116066 (+ 9174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:16.116073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:16.116078 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:16.116123 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:16.119572 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:16.120393 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:16.120398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:16.120402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:16.120543 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:16.120933 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:16.120937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:16.120940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:16.121263 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:16.121476 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:16.122025 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 23:22:16.122268 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:22:16.125283 7853 ts_tablet_manager.cc:1918] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:16.127485 7854 ts_tablet_manager.cc:1918] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:16.127700 7853 log.cc:1196] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/0d82a877d8ac46b3a09dd4f4c07bcde9
I20260620 23:22:16.128262 7853 ts_tablet_manager.cc:1939] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P d71f0e04eac14825a32248eaa74d2c86: Deleting consensus metadata
I20260620 23:22:16.129906 7854 log.cc:1196] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/0d82a877d8ac46b3a09dd4f4c07bcde9
I20260620 23:22:16.130448 7854 ts_tablet_manager.cc:1939] T 0d82a877d8ac46b3a09dd4f4c07bcde9 P 86483dafc79b4aa289bdafbcb739d07e: Deleting consensus metadata
I20260620 23:22:16.139846 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:16.094196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45816 (local address 127.0.97.60:37761)
0620 23:22:16.094387 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:16.094395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:16.124776 (+ 30381us) server_negotiation.cc:407] Connection header received
0620 23:22:16.124995 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:16.125001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:16.125076 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:16.125139 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:16.126798 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:16.127929 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:16.129725 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:16.130329 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:16.130418 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:16.133648 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:16.133654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:16.133659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:16.133694 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:16.137174 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:16.137902 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:16.137908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:16.137912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:16.138044 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:16.138436 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:16.138440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:16.138444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:16.138769 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:16.139022 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:16.139313 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 23:22:16.139512 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:22:16.222554 6526 catalog_manager.cc:4841] TS 86483dafc79b4aa289bdafbcb739d07e (127.0.97.1:40977): tablet 0d82a877d8ac46b3a09dd4f4c07bcde9 (table test-table [id=85ce1e0d97fc4c159efe4211a09dd640]) successfully deleted
I20260620 23:22:16.223004 6526 catalog_manager.cc:4841] TS d71f0e04eac14825a32248eaa74d2c86 (127.0.97.3:38711): tablet 0d82a877d8ac46b3a09dd4f4c07bcde9 (table test-table [id=85ce1e0d97fc4c159efe4211a09dd640]) successfully deleted
I20260620 23:22:16.223127 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 6509
I20260620 23:22:16.292621 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:16.265114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.0.97.61:45803)
0620 23:22:16.265320 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:16.265325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:16.274111 (+ 8786us) server_negotiation.cc:407] Connection header received
0620 23:22:16.274192 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:16.274197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:16.274279 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:16.274389 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:16.276866 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:16.278000 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:16.282758 (+ 4758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:16.283354 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:16.283479 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:16.286691 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:16.286698 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:16.286702 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:16.286738 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:16.289921 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:16.290701 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:16.290707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:16.290714 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:16.290827 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:16.291183 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:16.291188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:16.291191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:16.291539 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:16.291786 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:16.292096 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 23:22:16.292282 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:22:16.295393 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:16.274017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.0.97.60:37761)
0620 23:22:16.274698 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:16.274705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:16.277503 (+ 2798us) server_negotiation.cc:407] Connection header received
0620 23:22:16.277706 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:16.277711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:16.277784 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:16.277844 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:16.282803 (+ 4959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:16.283850 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:16.285715 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:16.286364 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:16.286487 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:16.289561 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:16.289567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:16.289572 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:16.289607 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:16.292754 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:16.293497 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:16.293502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:16.293505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:16.293641 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:16.294031 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:16.294036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:16.294040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:16.294295 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:16.294556 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:16.294889 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 23:22:16.295089 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493}
I20260620 23:22:17.067886 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.067119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.1:34055)
0620 23:22:17.067615 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.067728 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":283,"threads_started":1}
W20260620 23:22:17.068241 7509 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
I20260620 23:22:17.076454 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.075690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.2:48371)
0620 23:22:17.076107 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.076207 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":245,"threads_started":1}
W20260620 23:22:17.076809 7655 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
I20260620 23:22:17.080636 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.079807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.3:42121)
0620 23:22:17.080283 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.080408 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":287,"threads_started":1}
W20260620 23:22:17.080946 7796 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
I20260620 23:22:17.589423 7867 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c302f20b4e83403f9fe29b81eb2f7396)
I20260620 23:22:17.589741 7868 raft_consensus.cc:482] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c302f20b4e83403f9fe29b81eb2f7396)
I20260620 23:22:17.589692 7867 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } }
I20260620 23:22:17.589960 7868 raft_consensus.cc:504] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } }
I20260620 23:22:17.591033 7867 leader_election.cc:288] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331), df8d3de65ed749678f8a3fb4c3d20588 (127.0.97.60:37761)
I20260620 23:22:17.592244 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.591225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53598)
0620 23:22:17.591817 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.591977 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"threads_started":1}
I20260620 23:22:17.593279 7868 leader_election.cc:288] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331), d3316daf2d0b47188882ae176b98bf12 (127.0.97.61:45803)
I20260620 23:22:17.594431 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.591914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53606)
0620 23:22:17.594063 (+ 2149us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.594201 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2048,"thread_start_us":1905,"threads_started":1}
W20260620 23:22:17.594703 7315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
W20260620 23:22:17.595321 7124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
I20260620 23:22:17.597437 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.596938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53622)
0620 23:22:17.597244 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.597317 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
W20260620 23:22:17.597777 7315 leader_election.cc:334] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
I20260620 23:22:17.598335 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.597690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53624)
0620 23:22:17.598011 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.598152 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
W20260620 23:22:17.598495 7124 leader_election.cc:334] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
I20260620 23:22:17.600665 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.591263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37761 (local address 127.0.0.1:45826)
0620 23:22:17.592600 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.592622 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:17.592775 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:17.594217 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:17.594222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:17.594254 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:17.594635 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:17.594652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:17.596310 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:17.596318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:17.598589 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:17.598603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:17.599825 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:17.599831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:17.600044 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:17.600067 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:22:17.600175 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:22:17.600309 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1213,"thread_start_us":423,"threads_started":1}
I20260620 23:22:17.600685 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.591554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45826 (local address 127.0.97.60:37761)
0620 23:22:17.593895 (+ 2341us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:17.593907 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:17.593926 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:22:17.593991 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:17.593995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:17.594071 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:17.594179 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:17.594900 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:17.596180 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:17.598778 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:17.599662 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:17.599799 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:17.599890 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:17.600195 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 23:22:17.600314 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":2202,"thread_start_us":2031,"threads_started":1}
I20260620 23:22:17.601223 7339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3316daf2d0b47188882ae176b98bf12" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "df8d3de65ed749678f8a3fb4c3d20588" is_pre_election: true
I20260620 23:22:17.601496 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.591407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45803 (local address 127.0.0.1:59704)
0620 23:22:17.593351 (+ 1944us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.593392 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:17.593635 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:17.595111 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:17.595118 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:17.595154 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:17.595678 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:17.595696 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:17.597709 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:17.597715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:17.599541 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:17.599552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:17.600714 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:17.600720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:17.600892 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:17.600915 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:22:17.601025 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:22:17.601136 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1779,"mutex_wait_us":2,"thread_start_us":1612,"threads_started":1}
I20260620 23:22:17.601540 7339 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3316daf2d0b47188882ae176b98bf12 in term 1.
I20260620 23:22:17.601624 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.594193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.0.97.61:45803)
0620 23:22:17.594833 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:17.594839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:17.594856 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:22:17.594919 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:17.594923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:17.594981 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:17.595081 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:17.595925 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:17.597569 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:17.599684 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:17.600518 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:17.600677 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:17.600783 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:17.601179 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 23:22:17.601306 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":900,"threads_started":1}
I20260620 23:22:17.602059 7122 leader_election.cc:302] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3316daf2d0b47188882ae176b98bf12, df8d3de65ed749678f8a3fb4c3d20588; no voters: c302f20b4e83403f9fe29b81eb2f7396
I20260620 23:22:17.602139 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df8d3de65ed749678f8a3fb4c3d20588" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d3316daf2d0b47188882ae176b98bf12" is_pre_election: true
I20260620 23:22:17.602432 7148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8d3de65ed749678f8a3fb4c3d20588 in term 1.
I20260620 23:22:17.602609 7867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:22:17.602774 7867 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c302f20b4e83403f9fe29b81eb2f7396)
I20260620 23:22:17.602883 7867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:22:17.602903 7313 leader_election.cc:302] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3316daf2d0b47188882ae176b98bf12, df8d3de65ed749678f8a3fb4c3d20588; no voters: c302f20b4e83403f9fe29b81eb2f7396
I20260620 23:22:17.603227 7868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:22:17.603341 7868 raft_consensus.cc:482] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c302f20b4e83403f9fe29b81eb2f7396)
I20260620 23:22:17.603435 7868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:22:17.604924 7867 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } }
I20260620 23:22:17.605317 7868 raft_consensus.cc:504] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } }
I20260620 23:22:17.605546 7867 leader_election.cc:288] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [CANDIDATE]: Term 2 election: Requested vote from peers c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331), df8d3de65ed749678f8a3fb4c3d20588 (127.0.97.60:37761)
I20260620 23:22:17.605942 7868 leader_election.cc:288] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [CANDIDATE]: Term 2 election: Requested vote from peers c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331), d3316daf2d0b47188882ae176b98bf12 (127.0.97.61:45803)
I20260620 23:22:17.606091 7339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3316daf2d0b47188882ae176b98bf12" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "df8d3de65ed749678f8a3fb4c3d20588"
I20260620 23:22:17.606278 7339 raft_consensus.cc:2379] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d3316daf2d0b47188882ae176b98bf12 in current term 2: Already voted for candidate df8d3de65ed749678f8a3fb4c3d20588 in this term.
I20260620 23:22:17.606287 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df8d3de65ed749678f8a3fb4c3d20588" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d3316daf2d0b47188882ae176b98bf12"
I20260620 23:22:17.606474 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.605559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53636)
0620 23:22:17.605750 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.605826 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 23:22:17.606530 7148 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df8d3de65ed749678f8a3fb4c3d20588 in current term 2: Already voted for candidate d3316daf2d0b47188882ae176b98bf12 in this term.
I20260620 23:22:17.607326 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.607013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53652)
0620 23:22:17.607130 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.607233 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 23:22:17.607371 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.607057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53658)
0620 23:22:17.607216 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.607273 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 23:22:17.607528 7124 leader_election.cc:334] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
I20260620 23:22:17.607666 7124 leader_election.cc:302] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3316daf2d0b47188882ae176b98bf12; no voters: c302f20b4e83403f9fe29b81eb2f7396, df8d3de65ed749678f8a3fb4c3d20588
I20260620 23:22:17.607909 7867 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 23:22:17.608310 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:17.607940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53660)
0620 23:22:17.608149 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:17.608216 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260620 23:22:17.608472 7315 leader_election.cc:334] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
I20260620 23:22:17.608590 7315 leader_election.cc:302] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8d3de65ed749678f8a3fb4c3d20588; no voters: c302f20b4e83403f9fe29b81eb2f7396, d3316daf2d0b47188882ae176b98bf12
I20260620 23:22:17.608820 7868 raft_consensus.cc:2726] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 23:22:18.069468 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:18.068787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.1:33171)
0620 23:22:18.069231 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:18.069301 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":242,"threads_started":1}
I20260620 23:22:18.078074 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:18.077403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.2:35499)
0620 23:22:18.077838 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:18.077910 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":242,"threads_started":1}
I20260620 23:22:18.082304 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:18.081530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.3:57107)
0620 23:22:18.081967 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:18.082078 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":240,"threads_started":1}
I20260620 23:22:19.070991 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.070235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.1:34543)
0620 23:22:19.070771 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.070844 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":316,"threads_started":1}
I20260620 23:22:19.079416 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.078853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.2:51315)
0620 23:22:19.079204 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.079270 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":175,"threads_started":1}
I20260620 23:22:19.083699 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.083026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.3:45619)
0620 23:22:19.083446 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.083534 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":237,"threads_started":1}
I20260620 23:22:19.218453 7883 raft_consensus.cc:482] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:19.218654 7883 raft_consensus.cc:504] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } }
I20260620 23:22:19.219270 7883 leader_election.cc:288] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331), d3316daf2d0b47188882ae176b98bf12 (127.0.97.61:45803)
I20260620 23:22:19.219801 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df8d3de65ed749678f8a3fb4c3d20588" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d3316daf2d0b47188882ae176b98bf12" is_pre_election: true
I20260620 23:22:19.220084 7148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8d3de65ed749678f8a3fb4c3d20588 in term 2.
I20260620 23:22:19.220234 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.219417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53872)
0620 23:22:19.219915 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.220005 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":239,"threads_started":1}
I20260620 23:22:19.220548 7313 leader_election.cc:302] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3316daf2d0b47188882ae176b98bf12, df8d3de65ed749678f8a3fb4c3d20588; no voters:
I20260620 23:22:19.220821 7883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 23:22:19.220952 7883 raft_consensus.cc:482] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:22:19.221057 7883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:22:19.221582 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.221262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53882)
0620 23:22:19.221395 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.221465 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 23:22:19.221778 7315 leader_election.cc:334] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
I20260620 23:22:19.223666 7883 raft_consensus.cc:504] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } }
I20260620 23:22:19.224339 7883 leader_election.cc:288] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [CANDIDATE]: Term 3 election: Requested vote from peers c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331), d3316daf2d0b47188882ae176b98bf12 (127.0.97.61:45803)
I20260620 23:22:19.224726 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.224387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53896)
0620 23:22:19.224550 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.224628 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:22:19.224738 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df8d3de65ed749678f8a3fb4c3d20588" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d3316daf2d0b47188882ae176b98bf12"
I20260620 23:22:19.224959 7148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:22:19.225517 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.225264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53898)
0620 23:22:19.225372 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.225428 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 23:22:19.225708 7315 leader_election.cc:334] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
I20260620 23:22:19.227186 7148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8d3de65ed749678f8a3fb4c3d20588 in term 3.
I20260620 23:22:19.227548 7313 leader_election.cc:302] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3316daf2d0b47188882ae176b98bf12, df8d3de65ed749678f8a3fb4c3d20588; no voters: c302f20b4e83403f9fe29b81eb2f7396
I20260620 23:22:19.227809 7883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:22:19.228174 7883 raft_consensus.cc:686] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [term 3 LEADER]: Becoming Leader. State: Replica: df8d3de65ed749678f8a3fb4c3d20588, State: Running, Role: LEADER
I20260620 23:22:19.228521 7883 consensus_queue.cc:237] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } }
I20260620 23:22:19.230254 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8d3de65ed749678f8a3fb4c3d20588. Latest consensus state: current_term: 3 leader_uuid: "df8d3de65ed749678f8a3fb4c3d20588" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } } }
I20260620 23:22:19.230561 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:19.231149 7888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:19.234465 7888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:19.235246 7888 catalog_manager.cc:1259] Loaded cluster ID: a322d1c8cca648a09cabb7b34ccf8e82
I20260620 23:22:19.235339 7888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:19.236394 7888 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:19.237145 7888 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: Loaded TSK: 0
I20260620 23:22:19.237937 7888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:22:19.310792 7328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:22:19.317721 7328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:22:19.321475 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.321079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53924)
0620 23:22:19.321272 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.321342 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 23:22:19.321499 7148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [term 3 FOLLOWER]: Refusing update from remote peer df8d3de65ed749678f8a3fb4c3d20588: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 23:22:19.322196 7886 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 23:22:19.322430 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.322078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53928)
0620 23:22:19.322239 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.322312 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 23:22:19.322575 7315 consensus_peers.cc:487] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 -> Peer c302f20b4e83403f9fe29b81eb2f7396 (127.0.97.62:43331): Couldn't send request to peer c302f20b4e83403f9fe29b81eb2f7396. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:22:19.325223 7890 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8d3de65ed749678f8a3fb4c3d20588. Latest consensus state: current_term: 3 leader_uuid: "df8d3de65ed749678f8a3fb4c3d20588" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } } }
I20260620 23:22:19.325460 7890 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:22:19.326231 7883 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "df8d3de65ed749678f8a3fb4c3d20588" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } } }
I20260620 23:22:19.326596 7883 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:19.326730 7890 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "df8d3de65ed749678f8a3fb4c3d20588" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c302f20b4e83403f9fe29b81eb2f7396" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43331 } } peers { permanent_uuid: "d3316daf2d0b47188882ae176b98bf12" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45803 } attrs { promote: false } } peers { permanent_uuid: "df8d3de65ed749678f8a3fb4c3d20588" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37761 } attrs { promote: false } } }
I20260620 23:22:19.326975 7890 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3316daf2d0b47188882ae176b98bf12 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:22:19.343883 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.334144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40977 (local address 127.0.0.1:50752)
0620 23:22:19.334287 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.334309 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:19.334408 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:19.335639 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:19.335644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:19.335667 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:19.336029 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:19.336045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.337964 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.337972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:19.340593 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:19.340610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.342860 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.342868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:19.343112 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.343162 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 23:22:19.343279 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:22:19.343474 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 23:22:19.344161 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.335625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.0.97.3:38711)
0620 23:22:19.339057 (+ 3432us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:19.339061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:19.339074 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:22:19.339124 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:19.339127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:19.339182 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:19.339281 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:19.339796 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.340560 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.342266 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.343199 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.343372 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.343479 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:19.343731 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 23:22:19.343855 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3333,"thread_start_us":3234,"threads_started":1}
I20260620 23:22:19.344558 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.335313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38711 (local address 127.0.0.1:57164)
0620 23:22:19.335881 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.335902 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:19.336025 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:19.339323 (+ 3298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:19.339327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:19.339350 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:19.339645 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:19.339651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.340714 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.340719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:19.342111 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:19.342121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.343429 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.343435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:19.343591 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.343612 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:22:19.343710 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 23:22:19.343847 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":482,"threads_started":1}
I20260620 23:22:19.344861 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.334900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.0.97.1:40977)
0620 23:22:19.335314 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:19.335320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:19.335337 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:22:19.335409 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:19.335415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:19.335487 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:19.336198 (+ 711us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:19.336426 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.337779 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.340838 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.342435 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.342580 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.342738 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:19.344255 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0620 23:22:19.344462 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":832,"threads_started":1}
I20260620 23:22:19.345445 7730 tablet_service.cc:1467] Processing CreateTablet for tablet 363235aba3c64c188984690cd30ce432 (DEFAULT_TABLE table=test-table [id=81cc188d7a8344fba41e6f764d3ac6ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:19.345682 7443 tablet_service.cc:1467] Processing CreateTablet for tablet 363235aba3c64c188984690cd30ce432 (DEFAULT_TABLE table=test-table [id=81cc188d7a8344fba41e6f764d3ac6ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:19.345829 7730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 363235aba3c64c188984690cd30ce432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:19.346174 7443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 363235aba3c64c188984690cd30ce432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:19.351169 7901 tablet_bootstrap.cc:492] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86: Bootstrap starting.
I20260620 23:22:19.352324 7902 tablet_bootstrap.cc:492] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e: Bootstrap starting.
I20260620 23:22:19.353018 7901 tablet_bootstrap.cc:654] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:19.353749 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.334515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42345 (local address 127.0.0.1:35130)
0620 23:22:19.335106 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.335124 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:19.335219 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:19.347493 (+ 12274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:19.347497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:19.347520 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:19.347808 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:19.347818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.349122 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.349127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:19.351141 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:19.351153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.352891 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.352897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:19.353091 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.353130 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:22:19.353237 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 23:22:19.353370 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":504,"threads_started":1}
I20260620 23:22:19.353907 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.334847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35130 (local address 127.0.97.2:42345)
0620 23:22:19.347167 (+ 12320us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:19.347173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:19.347192 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:22:19.347255 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:19.347260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:19.347351 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:19.347457 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:19.347965 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.348998 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.351320 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.352541 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.352656 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.352761 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:19.353298 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 23:22:19.353492 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12193,"thread_start_us":12715,"threads_started":1}
I20260620 23:22:19.354071 7902 tablet_bootstrap.cc:654] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:19.354874 7589 tablet_service.cc:1467] Processing CreateTablet for tablet 363235aba3c64c188984690cd30ce432 (DEFAULT_TABLE table=test-table [id=81cc188d7a8344fba41e6f764d3ac6ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:19.355288 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 363235aba3c64c188984690cd30ce432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:19.356400 7901 tablet_bootstrap.cc:492] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86: No bootstrap required, opened a new log
I20260620 23:22:19.356576 7901 ts_tablet_manager.cc:1397] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 23:22:19.357563 7901 raft_consensus.cc:348] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:19.357874 7901 raft_consensus.cc:374] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:19.357982 7901 raft_consensus.cc:729] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71f0e04eac14825a32248eaa74d2c86, State: Initialized, Role: FOLLOWER
I20260620 23:22:19.358314 7901 consensus_queue.cc:260] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:19.359674 7901 ts_tablet_manager.cc:1428] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 23:22:19.360486 7902 tablet_bootstrap.cc:492] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e: No bootstrap required, opened a new log
I20260620 23:22:19.360615 7902 ts_tablet_manager.cc:1397] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 23:22:19.360637 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.360213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.3:38233)
0620 23:22:19.360391 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.360479 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 23:22:19.361624 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.361202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.3:37133)
0620 23:22:19.361394 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.361469 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 23:22:19.361603 7902 raft_consensus.cc:348] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:19.361861 7902 raft_consensus.cc:374] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:19.362026 7902 raft_consensus.cc:729] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86483dafc79b4aa289bdafbcb739d07e, State: Initialized, Role: FOLLOWER
I20260620 23:22:19.362522 7795 heartbeater.cc:503] Master 127.0.97.60:37761 was elected leader, sending a full tablet report...
I20260620 23:22:19.362694 7902 consensus_queue.cc:260] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:19.364571 7902 ts_tablet_manager.cc:1428] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260620 23:22:19.364785 7906 tablet_bootstrap.cc:492] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d: Bootstrap starting.
I20260620 23:22:19.366209 7906 tablet_bootstrap.cc:654] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:19.367180 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.366790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.1:55059)
0620 23:22:19.366964 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.367038 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 23:22:19.368352 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.367899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.1:41167)
0620 23:22:19.368112 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.368186 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 23:22:19.368645 7508 heartbeater.cc:503] Master 127.0.97.60:37761 was elected leader, sending a full tablet report...
I20260620 23:22:19.369833 7906 tablet_bootstrap.cc:492] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d: No bootstrap required, opened a new log
I20260620 23:22:19.369989 7906 ts_tablet_manager.cc:1397] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 23:22:19.370594 7906 raft_consensus.cc:348] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:19.370798 7906 raft_consensus.cc:374] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:19.370880 7906 raft_consensus.cc:729] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218b5570d52c4d6dab9617e51a26792d, State: Initialized, Role: FOLLOWER
I20260620 23:22:19.371230 7906 consensus_queue.cc:260] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:19.372699 7906 ts_tablet_manager.cc:1428] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:22:19.372949 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.372543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.2:35833)
0620 23:22:19.372703 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.372796 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 23:22:19.373924 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.373486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.2:41189)
0620 23:22:19.373634 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.373756 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:22:19.374064 7654 heartbeater.cc:503] Master 127.0.97.60:37761 was elected leader, sending a full tablet report...
W20260620 23:22:19.433970 7511 tablet.cc:2369] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:22:19.484607 7905 raft_consensus.cc:482] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:19.484849 7905 raft_consensus.cc:504] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:19.486269 7905 leader_election.cc:288] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86483dafc79b4aa289bdafbcb739d07e (127.0.97.1:40977), 218b5570d52c4d6dab9617e51a26792d (127.0.97.2:42345)
I20260620 23:22:19.492074 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.486636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41265 (local address 127.0.97.1:40977)
0620 23:22:19.486875 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:19.486882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:19.486903 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:22:19.487195 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:19.487200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:19.487278 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:19.487416 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:19.487917 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.488613 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.490327 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.491164 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.491305 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.491402 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:19.491679 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 23:22:19.491821 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:22:19.492090 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.486529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40977 (local address 127.0.97.3:41265)
0620 23:22:19.486739 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.486760 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:19.486879 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:19.487440 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:19.487445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:19.487475 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:19.487748 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:19.487758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.488753 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.488759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:19.490170 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:19.490181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.491355 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.491361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:19.491527 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.491551 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:22:19.491656 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:22:19.491822 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 23:22:19.492413 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.487048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42743 (local address 127.0.97.2:42345)
0620 23:22:19.487218 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:19.487223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:19.487704 (+ 481us) server_negotiation.cc:407] Connection header received
0620 23:22:19.487937 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:19.487942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:19.488006 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:19.488099 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:19.488564 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.489200 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.490781 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.491536 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.491679 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.491786 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:19.492053 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 23:22:19.492168 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:22:19.492456 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.486941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42345 (local address 127.0.97.3:42743)
0620 23:22:19.487589 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.487602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:19.487797 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:19.488114 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:19.488118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:19.488140 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:19.488407 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:19.488414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.489332 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.489339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:19.490612 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:19.490623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.491715 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.491721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:19.491905 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.491927 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:22:19.492032 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:22:19.492148 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":319,"threads_started":1}
I20260620 23:22:19.492846 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "363235aba3c64c188984690cd30ce432" candidate_uuid: "d71f0e04eac14825a32248eaa74d2c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86483dafc79b4aa289bdafbcb739d07e" is_pre_election: true
I20260620 23:22:19.493064 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "363235aba3c64c188984690cd30ce432" candidate_uuid: "d71f0e04eac14825a32248eaa74d2c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218b5570d52c4d6dab9617e51a26792d" is_pre_election: true
I20260620 23:22:19.493184 7463 raft_consensus.cc:2454] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d71f0e04eac14825a32248eaa74d2c86 in term 0.
I20260620 23:22:19.493291 7609 raft_consensus.cc:2454] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d71f0e04eac14825a32248eaa74d2c86 in term 0.
I20260620 23:22:19.493738 7684 leader_election.cc:302] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86483dafc79b4aa289bdafbcb739d07e, d71f0e04eac14825a32248eaa74d2c86; no voters:
I20260620 23:22:19.494071 7905 raft_consensus.cc:2781] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:22:19.494194 7905 raft_consensus.cc:482] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:22:19.494302 7905 raft_consensus.cc:3037] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:19.496374 7905 raft_consensus.cc:504] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:19.496924 7905 leader_election.cc:288] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [CANDIDATE]: Term 1 election: Requested vote from peers 86483dafc79b4aa289bdafbcb739d07e (127.0.97.1:40977), 218b5570d52c4d6dab9617e51a26792d (127.0.97.2:42345)
I20260620 23:22:19.497387 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "363235aba3c64c188984690cd30ce432" candidate_uuid: "d71f0e04eac14825a32248eaa74d2c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86483dafc79b4aa289bdafbcb739d07e"
I20260620 23:22:19.497442 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "363235aba3c64c188984690cd30ce432" candidate_uuid: "d71f0e04eac14825a32248eaa74d2c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218b5570d52c4d6dab9617e51a26792d"
I20260620 23:22:19.497571 7463 raft_consensus.cc:3037] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:19.497665 7609 raft_consensus.cc:3037] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:19.499630 7609 raft_consensus.cc:2454] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d71f0e04eac14825a32248eaa74d2c86 in term 1.
I20260620 23:22:19.499922 7463 raft_consensus.cc:2454] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d71f0e04eac14825a32248eaa74d2c86 in term 1.
I20260620 23:22:19.500041 7684 leader_election.cc:302] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218b5570d52c4d6dab9617e51a26792d, d71f0e04eac14825a32248eaa74d2c86; no voters:
I20260620 23:22:19.500316 7905 raft_consensus.cc:2781] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:19.500762 7905 raft_consensus.cc:686] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 1 LEADER]: Becoming Leader. State: Replica: d71f0e04eac14825a32248eaa74d2c86, State: Running, Role: LEADER
I20260620 23:22:19.501165 7905 consensus_queue.cc:237] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } }
I20260620 23:22:19.501761 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.501346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.3:43667)
0620 23:22:19.501536 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.501631 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 23:22:19.504752 7328 catalog_manager.cc:5495] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 reported cstate change: term changed from 0 to 1, leader changed from <none> to d71f0e04eac14825a32248eaa74d2c86 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d71f0e04eac14825a32248eaa74d2c86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d71f0e04eac14825a32248eaa74d2c86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38711 } health_report { overall_health: HEALTHY } } }
I20260620 23:22:19.531347 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.525921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.0.97.3:38711)
0620 23:22:19.526099 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:19.526104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:19.526122 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:22:19.526311 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:19.526314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:19.526390 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:19.526506 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:22:19.526899 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.527595 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.529403 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.529738 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.529816 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.530223 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:22:19.530798 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:22:19.530896 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:19.531018 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 23:22:19.531077 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:22:19.535070 7609 raft_consensus.cc:1261] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d [term 1 FOLLOWER]: Refusing update from remote peer d71f0e04eac14825a32248eaa74d2c86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:19.535492 7463 raft_consensus.cc:1261] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e [term 1 FOLLOWER]: Refusing update from remote peer d71f0e04eac14825a32248eaa74d2c86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:19.535765 7911 consensus_queue.cc:1035] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218b5570d52c4d6dab9617e51a26792d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:19.536226 7905 consensus_queue.cc:1035] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86483dafc79b4aa289bdafbcb739d07e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:19.538797 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.538279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.1:47955)
0620 23:22:19.538499 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.538593 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 23:22:19.539012 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.538500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.2:46117)
0620 23:22:19.538732 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.538845 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 23:22:19.543349 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.542811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.2:35799)
0620 23:22:19.543090 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.543168 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 23:22:19.544844 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.544381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.3:39735)
0620 23:22:19.544608 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.544697 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 23:22:19.545255 7915 mvcc.cc:205] Tried to move back new op lower bound from 7299062741129654272 to 7299062741000671232. Current Snapshot: MvccSnapshot[applied={T|T < 7299062741129654272}]
I20260620 23:22:19.553768 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.553133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.3:39965)
0620 23:22:19.553421 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.553527 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260620 23:22:19.560328 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.559798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.97.1:54151)
0620 23:22:19.560068 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.560145 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260620 23:22:19.560424 7329 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:19.560693 7329 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:19.562053 7916 mvcc.cc:205] Tried to move back new op lower bound from 7299062741129654272 to 7299062741000671232. Current Snapshot: MvccSnapshot[applied={T|T < 7299062741129654272}]
I20260620 23:22:19.568809 7443 tablet_service.cc:1514] Processing DeleteTablet for tablet 363235aba3c64c188984690cd30ce432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:19 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:50752
I20260620 23:22:19.568864 7329 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588: Sending DeleteTablet for 3 replicas of tablet 363235aba3c64c188984690cd30ce432
I20260620 23:22:19.570377 7730 tablet_service.cc:1514] Processing DeleteTablet for tablet 363235aba3c64c188984690cd30ce432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:19 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57164
I20260620 23:22:19.578512 7589 tablet_service.cc:1514] Processing DeleteTablet for tablet 363235aba3c64c188984690cd30ce432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:19 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:35130
I20260620 23:22:19.579227 7926 tablet_replica.cc:330] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d: stopping tablet replica
I20260620 23:22:19.579396 7926 raft_consensus.cc:2229] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:19.579696 7926 raft_consensus.cc:2258] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:19.579680 7442 tablet_service.cc:1514] Processing DeleteTablet for tablet 363235aba3c64c188984690cd30ce432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:19 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:50752
I20260620 23:22:19.580891 7926 ts_tablet_manager.cc:1905] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:19.581595 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 7301
I20260620 23:22:19.584923 7924 tablet_replica.cc:330] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e: stopping tablet replica
I20260620 23:22:19.585232 7924 raft_consensus.cc:2229] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:19.585614 7924 raft_consensus.cc:2258] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:19.587222 7924 ts_tablet_manager.cc:1905] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:19.592216 7926 ts_tablet_manager.cc:1918] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:19.592381 7926 log.cc:1196] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/363235aba3c64c188984690cd30ce432
I20260620 23:22:19.592847 7926 ts_tablet_manager.cc:1939] T 363235aba3c64c188984690cd30ce432 P 218b5570d52c4d6dab9617e51a26792d: Deleting consensus metadata
I20260620 23:22:19.593999 7314 catalog_manager.cc:4841] TS 218b5570d52c4d6dab9617e51a26792d (127.0.97.2:42345): tablet 363235aba3c64c188984690cd30ce432 (table test-table [id=81cc188d7a8344fba41e6f764d3ac6ab]) successfully deleted
I20260620 23:22:19.595834 7924 ts_tablet_manager.cc:1918] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:19.596035 7924 log.cc:1196] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/363235aba3c64c188984690cd30ce432
I20260620 23:22:19.596537 7924 ts_tablet_manager.cc:1939] T 363235aba3c64c188984690cd30ce432 P 86483dafc79b4aa289bdafbcb739d07e: Deleting consensus metadata
I20260620 23:22:19.598083 7925 tablet_replica.cc:330] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86: stopping tablet replica
I20260620 23:22:19.598311 7925 raft_consensus.cc:2229] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 1 LEADER]: Raft consensus shutting down.
W20260620 23:22:19.598460 7314 catalog_manager.cc:4808] TS 86483dafc79b4aa289bdafbcb739d07e (127.0.97.1:40977): delete failed for tablet 363235aba3c64c188984690cd30ce432 because the tablet was not found. No further retry: Not found: Tablet not found: 363235aba3c64c188984690cd30ce432
I20260620 23:22:19.599040 7925 raft_consensus.cc:2258] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:19.600049 7925 ts_tablet_manager.cc:1905] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:19.606562 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.582523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.0.97.1:40977)
0620 23:22:19.582721 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:19.582730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:19.592529 (+ 9799us) server_negotiation.cc:407] Connection header received
0620 23:22:19.592806 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:19.592813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:19.592896 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:19.592974 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:19.595025 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.596107 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.597915 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.598506 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.598612 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.601110 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:19.601121 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:19.601140 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:19.601171 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:19.604005 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:19.604600 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:19.604608 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:19.604612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:19.604737 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:19.605171 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:19.605176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:19.605179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:19.605489 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:19.605764 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:19.606022 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 23:22:19.606217 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:22:19.606580 7925 ts_tablet_manager.cc:1918] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:19.606714 7925 log.cc:1196] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/363235aba3c64c188984690cd30ce432
I20260620 23:22:19.607214 7503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:19.607228 7925 ts_tablet_manager.cc:1939] T 363235aba3c64c188984690cd30ce432 P d71f0e04eac14825a32248eaa74d2c86: Deleting consensus metadata
I20260620 23:22:19.608350 7314 catalog_manager.cc:4841] TS d71f0e04eac14825a32248eaa74d2c86 (127.0.97.3:38711): tablet 363235aba3c64c188984690cd30ce432 (table test-table [id=81cc188d7a8344fba41e6f764d3ac6ab]) successfully deleted
I20260620 23:22:19.724586 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 7301
I20260620 23:22:19.732892 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.732400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53930)
0620 23:22:19.732619 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.732728 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 23:22:19.734035 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.733655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43331 (local address 127.0.0.1:53940)
0620 23:22:19.733806 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:19.733890 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 23:22:19.746862 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 7520
I20260620 23:22:19.760242 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.747599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.0.97.2:42345)
0620 23:22:19.747808 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:19.747815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:19.747833 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:22:19.748009 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:19.748014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:19.748087 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:19.748193 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:19.749408 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.750139 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.751540 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.752095 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.752222 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.754746 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:19.754762 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:19.754786 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:19.754833 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:19.757592 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:19.758225 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:19.758233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:19.758237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:19.758364 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:19.758865 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:19.758869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:19.758870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:19.759192 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:19.759492 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:19.759723 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 23:22:19.759927 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:22:19.760939 7649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:19.881809 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 7520
I20260620 23:22:19.899742 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 7667
I20260620 23:22:19.915355 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:19.900497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.0.97.3:38711)
0620 23:22:19.900669 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:19.900673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:19.900724 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:22:19.900933 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:19.900937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:19.901006 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:19.901086 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:19.902884 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.903601 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.905339 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:19.905734 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:19.905847 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:19.909172 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:19.909189 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:19.909205 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:19.909249 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:19.912424 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:19.913187 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:19.913193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:19.913195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:19.913340 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:19.913765 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:19.913768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:19.913770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:19.914098 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:19.914321 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:19.914810 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 23:22:19.914974 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:22:19.916143 7790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:20.040048 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 7667
I20260620 23:22:20.060788 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 7002
I20260620 23:22:20.061821 7168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:20.171995 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 7002
W20260620 23:22:20.192078 7313 connection.cc:535] client connection to 127.0.97.61:45803 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:22:20.192406 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 7102
I20260620 23:22:20.193218 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:20.192794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45803 (local address 127.0.0.1:59714)
0620 23:22:20.192954 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:20.193032 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 23:22:20.193398 7359 generic_service.cc:196] Checking for leaks (request via RPC)
W20260620 23:22:20.193436 7313 consensus_peers.cc:487] T 00000000000000000000000000000000 P df8d3de65ed749678f8a3fb4c3d20588 -> Peer d3316daf2d0b47188882ae176b98bf12 (127.0.97.61:45803): Couldn't send request to peer d3316daf2d0b47188882ae176b98bf12. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:22:20.309473 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 7102
2026-06-20T23:22:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10637 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:22:20 dist-test-slave-5kzz krb5kdc[7935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:22:20 dist-test-slave-5kzz krb5kdc[7935](info): set up 2 sockets
Jun 20 23:22:20 dist-test-slave-5kzz krb5kdc[7935](info): commencing operation
krb5kdc: starting...
W20260620 23:22:21.371518 388 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:22:21 dist-test-slave-5kzz krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:22:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 23:22:21.561848 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42187
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:21.875618 7948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:21.875885 7948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:21.875972 7948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:21.882740 7948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:21.882834 7948 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:22:21.882877 7948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:21.882936 7948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:21.882982 7948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:21.883041 7948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:22:21.892318 7948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42187
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:21.893389 7948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:21.894593 7948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:21.905512 7956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:21.905481 7954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:21.905604 7953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:21.906288 7948 server_base.cc:1034] running on GCE node
I20260620 23:22:21.907037 7948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:21.908208 7948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:21.909452 7948 hybrid_clock.cc:648] HybridClock initialized: now 1781997741909404 us; error 47 us; skew 500 ppm
I20260620 23:22:21.909746 7948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:21 dist-test-slave-5kzz krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997741, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:21.912660 7948 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 23:22:21.914012 7948 webserver.cc:460] Webserver started at http://127.0.97.62:46607/ using document root <none> and password file <none>
I20260620 23:22:21.914613 7948 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:21.914762 7948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:21.915033 7948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:21.917673 7948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "85e41f399c0845f29a40ddb469796c26"
format_stamp: "Formatted at 2026-06-20 23:22:21 on dist-test-slave-5kzz"
server_key: "7d6214af12044c57b45ad4c66f41a9e1"
server_key_iv: "de74d625ef5ac43938a9ba9b8e7eea05"
server_key_version: "encryptionkey@0"
I20260620 23:22:21.918370 7948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "85e41f399c0845f29a40ddb469796c26"
format_stamp: "Formatted at 2026-06-20 23:22:21 on dist-test-slave-5kzz"
server_key: "7d6214af12044c57b45ad4c66f41a9e1"
server_key_iv: "de74d625ef5ac43938a9ba9b8e7eea05"
server_key_version: "encryptionkey@0"
I20260620 23:22:21.922955 7948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260620 23:22:21.925971 7963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:21.927174 7948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:21.927337 7948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "85e41f399c0845f29a40ddb469796c26"
format_stamp: "Formatted at 2026-06-20 23:22:21 on dist-test-slave-5kzz"
server_key: "7d6214af12044c57b45ad4c66f41a9e1"
server_key_iv: "de74d625ef5ac43938a9ba9b8e7eea05"
server_key_version: "encryptionkey@0"
I20260620 23:22:21.927562 7948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:21.941007 7948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:21.944873 7948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:21.945147 7948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:21.966440 7948 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42187
I20260620 23:22:21.966456 8024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42187 every 8 connection(s)
I20260620 23:22:21.967849 7948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:21.971261 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:21.978089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 7948
I20260620 23:22:21.978288 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:22:21.979117 388 external_mini_cluster.cc:1428] Setting key 57483e85382e667d9e70feec456b83cb
I20260620 23:22:21.980751 8025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26: Bootstrap starting.
I20260620 23:22:21.983771 8025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:21.985011 8025 log.cc:824] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:21.987684 8025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26: No bootstrap required, opened a new log
Jun 20 23:22:21 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:21.997437 8025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e41f399c0845f29a40ddb469796c26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42187 } }
I20260620 23:22:21.997861 8025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:21.997977 8025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e41f399c0845f29a40ddb469796c26, State: Initialized, Role: FOLLOWER
I20260620 23:22:21.998569 8025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e41f399c0845f29a40ddb469796c26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42187 } }
I20260620 23:22:21.998853 8025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:21.999007 8025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:21.999191 8025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:22.002215 8025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e41f399c0845f29a40ddb469796c26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42187 } }
I20260620 23:22:22.002507 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:21.980992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.0.97.62:42187)
0620 23:22:21.981710 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:21.981721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:21.981758 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:22:21.983492 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:21.983511 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:21.983850 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:21.984226 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:21.985984 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:21.987376 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:21.989116 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:21.989782 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:21.989906 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:21.994364 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:21.994374 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:21.994392 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:21.994443 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:21.998214 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:21.998968 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:21.998978 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:21.998988 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:21.999196 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:21.999560 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:21.999567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:21.999573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:22.000245 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:22.000554 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:22.000865 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 23:22:22.001271 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":357,"threads_started":1}
I20260620 23:22:22.002781 8025 leader_election.cc:302] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85e41f399c0845f29a40ddb469796c26; no voters:
I20260620 23:22:22.003367 8025 leader_election.cc:288] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:22:22.003494 8030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:22.005203 8030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [term 1 LEADER]: Becoming Leader. State: Replica: 85e41f399c0845f29a40ddb469796c26, State: Running, Role: LEADER
I20260620 23:22:22.005264 8025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:22.005744 8030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e41f399c0845f29a40ddb469796c26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42187 } }
I20260620 23:22:22.008613 8031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85e41f399c0845f29a40ddb469796c26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e41f399c0845f29a40ddb469796c26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42187 } } }
I20260620 23:22:22.009109 8031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:22.009032 8032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e41f399c0845f29a40ddb469796c26. Latest consensus state: current_term: 1 leader_uuid: "85e41f399c0845f29a40ddb469796c26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e41f399c0845f29a40ddb469796c26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42187 } } }
I20260620 23:22:22.009321 8032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:22.016124 8040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:22.021056 8040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:22.029433 8040 catalog_manager.cc:1347] Generated new cluster ID: 49b7e51c8ddf428489487181dcc46b87
I20260620 23:22:22.029546 8040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:22.036665 8040 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:22:22.038221 8040 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:22.046743 8040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26: Generated new TSK 0
I20260620 23:22:22.047500 8040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 23:22:22.106653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42187
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:22.436846 8051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:22.437242 8051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:22.437409 8051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:22.449721 8051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:22.449910 8051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:22.450055 8051 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:22:22.450201 8051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:22:22.465355 8051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:22.466670 8051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:22.468161 8051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:22.483225 8056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:22.483198 8057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:22.484031 8051 server_base.cc:1034] running on GCE node
W20260620 23:22:22.484195 8059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:22.484879 8051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:22.485672 8051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:22.486928 8051 hybrid_clock.cc:648] HybridClock initialized: now 1781997742486853 us; error 32 us; skew 500 ppm
I20260620 23:22:22.487221 8051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:22 dist-test-slave-5kzz krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997742, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:22.490058 8051 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 23:22:22.491478 8051 webserver.cc:460] Webserver started at http://127.0.97.1:33649/ using document root <none> and password file <none>
I20260620 23:22:22.492131 8051 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:22.492266 8051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:22.492590 8051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:22.495412 8051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "09933f6893dd454b94dbadfdf55768ed"
format_stamp: "Formatted at 2026-06-20 23:22:22 on dist-test-slave-5kzz"
server_key: "d42b47518accd9f1474c14adde630ddf"
server_key_iv: "10e2fa5f8663029e1d91d27ef7817a0c"
server_key_version: "encryptionkey@0"
I20260620 23:22:22.496191 8051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "09933f6893dd454b94dbadfdf55768ed"
format_stamp: "Formatted at 2026-06-20 23:22:22 on dist-test-slave-5kzz"
server_key: "d42b47518accd9f1474c14adde630ddf"
server_key_iv: "10e2fa5f8663029e1d91d27ef7817a0c"
server_key_version: "encryptionkey@0"
I20260620 23:22:22.500941 8051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 23:22:22.504139 8066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:22.505363 8051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:22:22.505523 8051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "09933f6893dd454b94dbadfdf55768ed"
format_stamp: "Formatted at 2026-06-20 23:22:22 on dist-test-slave-5kzz"
server_key: "d42b47518accd9f1474c14adde630ddf"
server_key_iv: "10e2fa5f8663029e1d91d27ef7817a0c"
server_key_version: "encryptionkey@0"
I20260620 23:22:22.505750 8051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:22.523213 8051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:22.527319 8051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:22.527570 8051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:22.529937 8051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:22.530030 8051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:22.530145 8051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:22.530225 8051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:22:22 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997742, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:22.552121 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:22.530565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55745 (local address 127.0.97.62:42187)
0620 23:22:22.531165 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:22.531170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:22.532301 (+ 1131us) server_negotiation.cc:407] Connection header received
0620 23:22:22.534270 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:22.534275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:22.534368 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:22.534494 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:22.536670 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:22.537680 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:22.540808 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:22.541268 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:22.541362 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:22.545929 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:22.545935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:22.545939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:22.545977 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:22.549130 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:22.549789 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:22.549798 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:22.549800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:22.549907 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:22.550296 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:22.550301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:22.550304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:22.550551 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:22.550738 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:22.551579 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 23:22:22.551781 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":404,"threads_started":1}
I20260620 23:22:22.553086 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:22.531169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42187 (local address 127.0.97.1:55745)
0620 23:22:22.532138 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:22.532186 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:22.534003 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:22.534824 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:22.534839 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:22.535642 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:22.536492 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:22.536511 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:22.537887 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:22.537896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:22.540638 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:22.540657 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:22.541484 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:22.541494 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:22.541881 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:22.542807 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:22.542855 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:22.545685 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:22.549264 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:22.549272 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:22.549287 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:22.549668 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:22.550010 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:22.550016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:22.550020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:22.550186 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:22.550721 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:22.550729 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:22.551120 (+ 391us) client_negotiation.cc:769] Sending connection context
0620 23:22:22.551499 (+ 379us) client_negotiation.cc:240] Negotiation successful
0620 23:22:22.551997 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":462,"threads_started":1}
I20260620 23:22:22.573982 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:22.559387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42187 (local address 127.0.97.1:60827)
0620 23:22:22.560052 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:22.560067 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:22.560163 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:22.560517 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:22.560520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:22.560899 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:22.561161 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:22.561168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:22.562815 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:22.562818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:22.563749 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:22.563759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:22.564865 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:22.564869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:22.565004 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:22.565645 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:22.565673 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:22.566649 (+ 976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:22.570420 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:22.570426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:22.570435 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:22.571155 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:22.571569 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:22.571574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:22.571578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:22.571731 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:22.572454 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:22.572460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:22.572691 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 23:22:22.572865 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 23:22:22.573290 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":348,"threads_started":1}
I20260620 23:22:22.574693 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:22.559353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60827 (local address 127.0.97.62:42187)
0620 23:22:22.559577 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:22.559586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:22.560141 (+ 555us) server_negotiation.cc:407] Connection header received
0620 23:22:22.560268 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:22.560276 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:22.560382 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:22.560509 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:22.561287 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:22.562671 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:22.563872 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:22.564704 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:22.564847 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:22.566884 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:22.566892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:22.566899 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:22.566938 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:22.570264 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:22.571298 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:22.571306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:22.571311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:22.571469 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:22.571848 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:22.571856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:22.571862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:22.572229 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:22.572393 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:22.574012 (+ 1619us) server_negotiation.cc:299] Negotiation successful
0620 23:22:22.574232 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:22:22.584079 8051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40741
I20260620 23:22:22.584121 8186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40741 every 8 connection(s)
I20260620 23:22:22.585196 8051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:22:22.587080 8187 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42187
I20260620 23:22:22.587288 8187 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:22.587896 8187 heartbeater.cc:511] Master 127.0.97.62:42187 requested a full tablet report, sending...
I20260620 23:22:22.589331 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 8051
I20260620 23:22:22.589514 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:22:22.589708 7980 ts_manager.cc:194] Registered new tserver with Master: 09933f6893dd454b94dbadfdf55768ed (127.0.97.1:40741)
I20260620 23:22:22.590142 388 external_mini_cluster.cc:1428] Setting key fe016d7ba0e6f3db6d663e87f44927f5
I20260620 23:22:22.591560 7980 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55745
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 23:22:22.631605 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42187
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:22.935168 8191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:22.935402 8191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:22.935501 8191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:22.942245 8191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:22.942340 8191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:22.942420 8191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:22:22.942494 8191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:22:22.952603 8191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:22.953730 8191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:22.954993 8191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:22.967453 8196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:22.967769 8199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:22.967517 8197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:22.968894 8191 server_base.cc:1034] running on GCE node
I20260620 23:22:22.969415 8191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:22.970249 8191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:22.971477 8191 hybrid_clock.cc:648] HybridClock initialized: now 1781997742971410 us; error 38 us; skew 500 ppm
I20260620 23:22:22.971742 8191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:22 dist-test-slave-5kzz krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997742, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:22.974670 8191 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 23:22:22.976061 8191 webserver.cc:460] Webserver started at http://127.0.97.2:43093/ using document root <none> and password file <none>
I20260620 23:22:22.976689 8191 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:22.976809 8191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:22.977104 8191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:22.979890 8191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "540297eeb08149119ac1aca54f874888"
format_stamp: "Formatted at 2026-06-20 23:22:22 on dist-test-slave-5kzz"
server_key: "88210e494dedf28732b8aa09277be25e"
server_key_iv: "292f269e926e2fc8541affd70346a7d8"
server_key_version: "encryptionkey@0"
I20260620 23:22:22.980629 8191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "540297eeb08149119ac1aca54f874888"
format_stamp: "Formatted at 2026-06-20 23:22:22 on dist-test-slave-5kzz"
server_key: "88210e494dedf28732b8aa09277be25e"
server_key_iv: "292f269e926e2fc8541affd70346a7d8"
server_key_version: "encryptionkey@0"
I20260620 23:22:22.985344 8191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:22:22.988524 8206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:22.989715 8191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:22:22.989877 8191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "540297eeb08149119ac1aca54f874888"
format_stamp: "Formatted at 2026-06-20 23:22:22 on dist-test-slave-5kzz"
server_key: "88210e494dedf28732b8aa09277be25e"
server_key_iv: "292f269e926e2fc8541affd70346a7d8"
server_key_version: "encryptionkey@0"
I20260620 23:22:22.990109 8191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:23.007205 8191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:23.011217 8191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:23.011463 8191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:23.014402 8191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:23.014547 8191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:23.014739 8191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:23.014837 8191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:22:23 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997742, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:23.043877 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.016228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52967 (local address 127.0.97.62:42187)
0620 23:22:23.016828 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:23.016837 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:23.017809 (+ 972us) server_negotiation.cc:407] Connection header received
0620 23:22:23.021309 (+ 3500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:23.021317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:23.021424 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:23.021519 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:23.024739 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.026007 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.029316 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.030075 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.030201 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.035854 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:23.035860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:23.035865 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:23.035906 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:23.039728 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:23.040878 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:23.040886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:23.040891 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:23.041039 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:23.041718 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:23.041726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:23.041731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:23.042107 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:23.042267 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:23.043294 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0620 23:22:23.043533 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260620 23:22:23.045636 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.015492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42187 (local address 127.0.97.2:52967)
0620 23:22:23.017611 (+ 2119us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:23.017663 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:23.020934 (+ 3271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:23.022132 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:23.022148 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:23.023171 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:23.024492 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.024528 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.026232 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.026240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.029120 (+ 2880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.029139 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.030615 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.030628 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.031047 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.031892 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:23.031927 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:23.035553 (+ 3626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:23.039895 (+ 4342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:23.039909 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:23.039932 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:23.040708 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:23.041187 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:23.041191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:23.041198 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:23.041560 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:23.042344 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:23.042351 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:23.042854 (+ 503us) client_negotiation.cc:769] Sending connection context
0620 23:22:23.043183 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 23:22:23.043832 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1793,"thread_start_us":1552,"threads_started":1}
I20260620 23:22:23.063321 8191 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41001
I20260620 23:22:23.063335 8325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41001 every 8 connection(s)
I20260620 23:22:23.064774 8191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:22:23.065196 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.051211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37767 (local address 127.0.97.62:42187)
0620 23:22:23.051382 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:23.051388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:23.052334 (+ 946us) server_negotiation.cc:407] Connection header received
0620 23:22:23.052530 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:23.052534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:23.052589 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:23.052666 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:23.054048 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.054807 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.056828 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.057215 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.057349 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.059388 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:23.059397 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:23.059405 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:23.059466 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:23.062540 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:23.063535 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:23.063540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:23.063543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:23.063675 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:23.064026 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:23.064029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:23.064031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:23.064312 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:23.064425 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:23.064849 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 23:22:23.064998 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 23:22:23.065755 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.051448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42187 (local address 127.0.97.2:37767)
0620 23:22:23.052197 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:23.052219 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:23.052399 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:23.052743 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:23.052751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:23.053431 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:23.053892 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.053907 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.054974 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.054979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.056692 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.056704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.057394 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.057400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.057554 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.058219 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:23.058236 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:23.059022 (+ 786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:23.062691 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:23.062700 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:23.062706 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:23.063337 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:23.063779 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:23.063784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:23.063788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:23.063927 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:23.064443 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:23.064448 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:23.064624 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 23:22:23.064795 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 23:22:23.065095 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":413,"threads_started":1}
I20260620 23:22:23.067204 8326 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42187
I20260620 23:22:23.067480 8326 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:23.068140 8326 heartbeater.cc:511] Master 127.0.97.62:42187 requested a full tablet report, sending...
I20260620 23:22:23.069769 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 8191
I20260620 23:22:23.069932 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:22:23.069957 7980 ts_manager.cc:194] Registered new tserver with Master: 540297eeb08149119ac1aca54f874888 (127.0.97.2:41001)
I20260620 23:22:23.070451 388 external_mini_cluster.cc:1428] Setting key a20b246367c7d8ad189280230d51c874
I20260620 23:22:23.070999 7980 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52967
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 23:22:23.111042 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42187
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:23.418493 8330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:23.418813 8330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:23.418972 8330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:23.425765 8330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:23.425900 8330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:23.426043 8330 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:22:23.426173 8330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:22:23.435556 8330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:23.436702 8330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:23.438005 8330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:23.450461 8336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:23.450556 8335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:23.450769 8338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:23.451098 8330 server_base.cc:1034] running on GCE node
I20260620 23:22:23.451669 8330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:23.452477 8330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:23.453704 8330 hybrid_clock.cc:648] HybridClock initialized: now 1781997743453641 us; error 43 us; skew 500 ppm
I20260620 23:22:23.454005 8330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:23 dist-test-slave-5kzz krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997743, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:23.457065 8330 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 23:22:23.458416 8330 webserver.cc:460] Webserver started at http://127.0.97.3:35053/ using document root <none> and password file <none>
I20260620 23:22:23.459149 8330 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:23.459227 8330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:23.459471 8330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:23.462030 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "eb43c4a372264ba7a02626c6feb31ca6"
format_stamp: "Formatted at 2026-06-20 23:22:23 on dist-test-slave-5kzz"
server_key: "50637840701ed8fd4413c7faab08e82f"
server_key_iv: "7235a28a49aa6a37023616845e024dc7"
server_key_version: "encryptionkey@0"
I20260620 23:22:23.462755 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "eb43c4a372264ba7a02626c6feb31ca6"
format_stamp: "Formatted at 2026-06-20 23:22:23 on dist-test-slave-5kzz"
server_key: "50637840701ed8fd4413c7faab08e82f"
server_key_iv: "7235a28a49aa6a37023616845e024dc7"
server_key_version: "encryptionkey@0"
I20260620 23:22:23.467454 8330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:22:23.470527 8345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:23.471715 8330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 23:22:23.471846 8330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "eb43c4a372264ba7a02626c6feb31ca6"
format_stamp: "Formatted at 2026-06-20 23:22:23 on dist-test-slave-5kzz"
server_key: "50637840701ed8fd4413c7faab08e82f"
server_key_iv: "7235a28a49aa6a37023616845e024dc7"
server_key_version: "encryptionkey@0"
I20260620 23:22:23.472005 8330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:23.497092 8330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:23.501045 8330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:23.501245 8330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:23.503789 8330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:23.503890 8330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:23.504056 8330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:23.504127 8330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:22:23 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997743, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:23.524858 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.503986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41673 (local address 127.0.97.62:42187)
0620 23:22:23.504178 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:23.504186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:23.505500 (+ 1314us) server_negotiation.cc:407] Connection header received
0620 23:22:23.507404 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:23.507409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:23.507539 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:23.507667 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:23.509702 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.510525 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.513147 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.513952 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.514104 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.518035 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:23.518044 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:23.518052 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:23.518104 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:23.521588 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:23.522258 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:23.522263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:23.522267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:23.522387 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:23.522912 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:23.522917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:23.522922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:23.523221 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:23.523336 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:23.524466 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0620 23:22:23.524619 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:22:23.526778 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.504519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42187 (local address 127.0.97.3:41673)
0620 23:22:23.505338 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:23.505387 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:23.507161 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:23.507918 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:23.507926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:23.508509 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:23.509523 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.509549 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.510716 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.510723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.512913 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.512923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.514168 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.514174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.514350 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.515047 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:23.515084 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:23.517819 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:23.521715 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:23.521724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:23.521738 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:23.522137 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:23.522496 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:23.522503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:23.522507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:23.522783 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:23.523368 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:23.523379 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:23.523909 (+ 530us) client_negotiation.cc:769] Sending connection context
0620 23:22:23.524372 (+ 463us) client_negotiation.cc:240] Negotiation successful
0620 23:22:23.525020 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":326,"threads_started":1}
I20260620 23:22:23.547222 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.533040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42187 (local address 127.0.97.3:58515)
0620 23:22:23.533617 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:23.533635 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:23.533761 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:23.534113 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:23.534116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:23.534489 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:23.534824 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.534834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.536148 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.536153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.537849 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.537860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.538673 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.538720 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.538905 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.539605 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:23.539651 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:23.540641 (+ 990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:23.544594 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:23.544600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:23.544607 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:23.544974 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:23.545379 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:23.545382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:23.545384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:23.545497 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:23.546146 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:23.546152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:23.546345 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 23:22:23.546496 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 23:22:23.546784 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":419,"threads_started":1}
I20260620 23:22:23.547407 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.533397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58515 (local address 127.0.97.62:42187)
0620 23:22:23.533620 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:23.533629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:23.533718 (+ 89us) server_negotiation.cc:407] Connection header received
0620 23:22:23.533877 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:23.533885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:23.533977 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:23.534103 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:23.534958 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.535983 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.538002 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.538498 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.538635 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.540844 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:23.540852 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:23.540858 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:23.540899 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:23.544465 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:23.545081 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:23.545090 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:23.545095 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:23.545262 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:23.545592 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:23.545599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:23.545605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:23.546019 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:23.546186 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:23.546661 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 23:22:23.546962 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:22:23.558800 8330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40951
I20260620 23:22:23.558874 8464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40951 every 8 connection(s)
I20260620 23:22:23.560002 8330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:22:23.561762 8465 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42187
I20260620 23:22:23.561980 8465 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:23.562538 8465 heartbeater.cc:511] Master 127.0.97.62:42187 requested a full tablet report, sending...
I20260620 23:22:23.563753 7980 ts_manager.cc:194] Registered new tserver with Master: eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951)
I20260620 23:22:23.564527 7980 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41673
I20260620 23:22:23.569006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 8330
I20260620 23:22:23.569162 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:22:23.569567 388 external_mini_cluster.cc:1428] Setting key 7a49526a5a34f2d76e39edd08122c205
I20260620 23:22:23.572641 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:22:23.594058 8187 heartbeater.cc:503] Master 127.0.97.62:42187 was elected leader, sending a full tablet report...
Jun 20 23:22:23 dist-test-slave-5kzz krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:22:23 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:23 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:22:23.620338 8075 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:23.610799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.0.97.62:42187)
0620 23:22:23.610962 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:23.610967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:23.611320 (+ 353us) server_negotiation.cc:407] Connection header received
0620 23:22:23.611458 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:23.611462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:23.611520 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:23.611606 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:23.612803 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.613480 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.614749 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.615148 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.615239 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.620148 (+ 4909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43288: BlockingRecv error: recv got EOF from 127.0.0.1:43288 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 23:22:23 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:23.635435 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.623120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.0.97.62:42187)
0620 23:22:23.623305 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:23.623310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:23.623541 (+ 231us) server_negotiation.cc:407] Connection header received
0620 23:22:23.623721 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:23.623726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:23.623801 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:23.623895 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:23.624914 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.625595 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.626670 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.627089 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.627204 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.630645 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:23.630649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:23.630652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:23.630719 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:23.633303 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:23.633860 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:23.633863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:23.633866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:23.633981 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:23.634265 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:23.634268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:23.634270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:23.634500 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:23.634765 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:23.634972 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 23:22:23.635118 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:22:23.638862 7980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:22:23.645394 7980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:22:23.677287 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.655922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.0.1:55258)
0620 23:22:23.666653 (+ 10731us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:23.666714 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:23.666875 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:23.668911 (+ 2036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:23.668924 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:23.668968 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:23.669520 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.669539 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.671702 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.671707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.674540 (+ 2833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.674554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.676406 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.676412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.676583 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.676640 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 23:22:23.676785 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 23:22:23.676968 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10473,"thread_start_us":10328,"threads_started":1}
I20260620 23:22:23.677269 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.656114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.0.97.1:40741)
0620 23:22:23.666218 (+ 10104us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:23.666231 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:23.667056 (+ 825us) server_negotiation.cc:407] Connection header received
0620 23:22:23.668270 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:23.668282 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:23.668616 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:23.669780 (+ 1164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:23.669959 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.671552 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.674712 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.676208 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.676369 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.676584 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:23.676829 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 23:22:23.676981 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9854,"thread_start_us":9708,"threads_started":1}
I20260620 23:22:23.678943 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.670093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41001 (local address 127.0.0.1:44974)
0620 23:22:23.670880 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:23.670896 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:23.671017 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:23.671558 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:23.671563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:23.671595 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:23.672073 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.672084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.673577 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.673585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.675732 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.675747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.677883 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.677889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.678306 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.678335 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:22:23.678423 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:22:23.678565 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":317,"thread_start_us":540,"threads_started":1}
I20260620 23:22:23.679174 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.656707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.0.97.3:40951)
0620 23:22:23.670646 (+ 13939us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:23.670657 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:23.670714 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:22:23.670818 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:23.670826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:23.671053 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:23.671226 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:23.671839 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.673482 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.675873 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.677270 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.677390 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.677629 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:23.678700 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0620 23:22:23.678857 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13777,"thread_start_us":13610,"threads_started":1}
I20260620 23:22:23.678563 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.656081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40951 (local address 127.0.0.1:42952)
0620 23:22:23.669119 (+ 13038us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:23.669143 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:23.669297 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:23.671213 (+ 1916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:23.671217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:23.671239 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:23.671627 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.671639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.673657 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.673662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.675650 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.675668 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.677664 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.677671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.677867 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.677898 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:22:23.677989 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 23:22:23.678137 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12865,"mutex_wait_us":30,"thread_start_us":13356,"threads_started":1}
I20260620 23:22:23.679587 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.667304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.0.97.2:41001)
0620 23:22:23.670101 (+ 2797us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:23.670113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:23.671011 (+ 898us) server_negotiation.cc:407] Connection header received
0620 23:22:23.671160 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:23.671167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:23.671352 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:23.671530 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:23.672293 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.673422 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.675873 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.677270 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.678069 (+ 799us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.678237 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:23.679171 (+ 934us) server_negotiation.cc:299] Negotiation successful
0620 23:22:23.679320 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2612,"thread_start_us":2472,"threads_started":1}
I20260620 23:22:23.681509 8117 tablet_service.cc:1467] Processing CreateTablet for tablet f8e0bcc304dd4ee294b9cda16d710320 (DEFAULT_TABLE table=test-table [id=84caa151d4db40d5a3cc8bfe5fe4b38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:23.682921 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8e0bcc304dd4ee294b9cda16d710320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:23.682647 8395 tablet_service.cc:1467] Processing CreateTablet for tablet f8e0bcc304dd4ee294b9cda16d710320 (DEFAULT_TABLE table=test-table [id=84caa151d4db40d5a3cc8bfe5fe4b38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:23.682787 8256 tablet_service.cc:1467] Processing CreateTablet for tablet f8e0bcc304dd4ee294b9cda16d710320 (DEFAULT_TABLE table=test-table [id=84caa151d4db40d5a3cc8bfe5fe4b38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:23.683898 8256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8e0bcc304dd4ee294b9cda16d710320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:23.684023 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8e0bcc304dd4ee294b9cda16d710320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:23.695406 8489 tablet_bootstrap.cc:492] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888: Bootstrap starting.
I20260620 23:22:23.697469 8491 tablet_bootstrap.cc:492] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6: Bootstrap starting.
I20260620 23:22:23.699076 8489 tablet_bootstrap.cc:654] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:23.699301 8490 tablet_bootstrap.cc:492] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed: Bootstrap starting.
I20260620 23:22:23.700479 8489 log.cc:824] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:23.701232 8491 tablet_bootstrap.cc:654] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:23.702167 8490 tablet_bootstrap.cc:654] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:23.702337 8491 log.cc:824] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:23.703299 8490 log.cc:824] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:23.703474 8489 tablet_bootstrap.cc:492] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888: No bootstrap required, opened a new log
I20260620 23:22:23.703755 8489 ts_tablet_manager.cc:1397] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 23:22:23.705231 8491 tablet_bootstrap.cc:492] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6: No bootstrap required, opened a new log
I20260620 23:22:23.705564 8491 ts_tablet_manager.cc:1397] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260620 23:22:23.706065 8490 tablet_bootstrap.cc:492] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed: No bootstrap required, opened a new log
I20260620 23:22:23.706307 8490 ts_tablet_manager.cc:1397] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 23:22:23.713409 8490 raft_consensus.cc:348] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } }
I20260620 23:22:23.713770 8490 raft_consensus.cc:374] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:23.713896 8490 raft_consensus.cc:729] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09933f6893dd454b94dbadfdf55768ed, State: Initialized, Role: FOLLOWER
I20260620 23:22:23.714473 8490 consensus_queue.cc:260] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } }
I20260620 23:22:23.714609 8489 raft_consensus.cc:348] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } }
I20260620 23:22:23.715087 8489 raft_consensus.cc:374] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:23.715210 8489 raft_consensus.cc:729] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540297eeb08149119ac1aca54f874888, State: Initialized, Role: FOLLOWER
I20260620 23:22:23.715853 8489 consensus_queue.cc:260] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } }
I20260620 23:22:23.715802 8491 raft_consensus.cc:348] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } }
I20260620 23:22:23.716060 8490 ts_tablet_manager.cc:1428] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260620 23:22:23.716190 8491 raft_consensus.cc:374] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:23.716318 8491 raft_consensus.cc:729] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb43c4a372264ba7a02626c6feb31ca6, State: Initialized, Role: FOLLOWER
I20260620 23:22:23.717015 8491 consensus_queue.cc:260] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } }
I20260620 23:22:23.719930 8465 heartbeater.cc:503] Master 127.0.97.62:42187 was elected leader, sending a full tablet report...
I20260620 23:22:23.720148 8491 ts_tablet_manager.cc:1428] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 23:22:23.721856 8326 heartbeater.cc:503] Master 127.0.97.62:42187 was elected leader, sending a full tablet report...
I20260620 23:22:23.722455 8489 ts_tablet_manager.cc:1428] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
I20260620 23:22:23.804006 8495 raft_consensus.cc:482] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:23.804319 8495 raft_consensus.cc:504] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } }
I20260620 23:22:23.805804 8495 leader_election.cc:288] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951), 540297eeb08149119ac1aca54f874888 (127.0.97.2:41001)
W20260620 23:22:23.811494 8466 tablet.cc:2369] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:22:23.813442 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.806363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37189 (local address 127.0.97.3:40951)
0620 23:22:23.806527 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:23.806533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:23.807644 (+ 1111us) server_negotiation.cc:407] Connection header received
0620 23:22:23.807786 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:23.807791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:23.807866 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:23.807986 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:23.808686 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.809414 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.811359 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.812564 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.812668 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.812767 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:23.813081 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 23:22:23.813201 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:22:23.813467 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.806155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57071 (local address 127.0.97.2:41001)
0620 23:22:23.806354 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:23.806361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:23.807211 (+ 850us) server_negotiation.cc:407] Connection header received
0620 23:22:23.807488 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:23.807497 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:23.807585 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:23.807671 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:23.808469 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.809357 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.811272 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.812422 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.812563 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.812679 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:23.813082 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 23:22:23.813201 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:22:23.813737 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.806146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40951 (local address 127.0.97.1:37189)
0620 23:22:23.807537 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:23.807554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:23.807672 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:23.808002 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:23.808006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:23.808037 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:23.808533 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.808542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.809574 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.809581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.811194 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.811205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.812736 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.812741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.812944 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.812970 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:22:23.813044 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 23:22:23.813221 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1250,"mutex_wait_us":80,"thread_start_us":541,"threads_started":1}
I20260620 23:22:23.813759 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.806085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41001 (local address 127.0.97.1:57071)
0620 23:22:23.807094 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:23.807130 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:23.807333 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:23.807814 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:23.807819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:23.807845 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:23.808206 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.808214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.809468 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.809474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.811127 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:23.811138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.812648 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.812654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:23.812845 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.812937 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 23:22:23.813036 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:22:23.813221 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":69,"thread_start_us":1129,"threads_started":1}
I20260620 23:22:23.814668 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8e0bcc304dd4ee294b9cda16d710320" candidate_uuid: "09933f6893dd454b94dbadfdf55768ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb43c4a372264ba7a02626c6feb31ca6" is_pre_election: true
I20260620 23:22:23.814805 8276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8e0bcc304dd4ee294b9cda16d710320" candidate_uuid: "09933f6893dd454b94dbadfdf55768ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540297eeb08149119ac1aca54f874888" is_pre_election: true
I20260620 23:22:23.815246 8420 raft_consensus.cc:2454] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09933f6893dd454b94dbadfdf55768ed in term 0.
I20260620 23:22:23.815253 8276 raft_consensus.cc:2454] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09933f6893dd454b94dbadfdf55768ed in term 0.
I20260620 23:22:23.815830 8069 leader_election.cc:302] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09933f6893dd454b94dbadfdf55768ed, eb43c4a372264ba7a02626c6feb31ca6; no voters:
I20260620 23:22:23.816213 8495 raft_consensus.cc:2781] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:22:23.816356 8495 raft_consensus.cc:482] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:22:23.816453 8495 raft_consensus.cc:3037] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:22:23.817883 8327 tablet.cc:2369] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:22:23.818990 8495 raft_consensus.cc:504] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } }
I20260620 23:22:23.819563 8495 leader_election.cc:288] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [CANDIDATE]: Term 1 election: Requested vote from peers eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951), 540297eeb08149119ac1aca54f874888 (127.0.97.2:41001)
I20260620 23:22:23.819909 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8e0bcc304dd4ee294b9cda16d710320" candidate_uuid: "09933f6893dd454b94dbadfdf55768ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb43c4a372264ba7a02626c6feb31ca6"
I20260620 23:22:23.820022 8276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8e0bcc304dd4ee294b9cda16d710320" candidate_uuid: "09933f6893dd454b94dbadfdf55768ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540297eeb08149119ac1aca54f874888"
I20260620 23:22:23.820114 8420 raft_consensus.cc:3037] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:23.820199 8276 raft_consensus.cc:3037] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:23.823273 8276 raft_consensus.cc:2454] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09933f6893dd454b94dbadfdf55768ed in term 1.
I20260620 23:22:23.823273 8420 raft_consensus.cc:2454] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09933f6893dd454b94dbadfdf55768ed in term 1.
I20260620 23:22:23.823655 8067 leader_election.cc:302] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09933f6893dd454b94dbadfdf55768ed, 540297eeb08149119ac1aca54f874888; no voters:
I20260620 23:22:23.823967 8495 raft_consensus.cc:2781] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:23.824410 8495 raft_consensus.cc:686] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 1 LEADER]: Becoming Leader. State: Replica: 09933f6893dd454b94dbadfdf55768ed, State: Running, Role: LEADER
I20260620 23:22:23.824903 8495 consensus_queue.cc:237] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } }
I20260620 23:22:23.829470 7979 catalog_manager.cc:5495] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 09933f6893dd454b94dbadfdf55768ed (127.0.97.1). New cstate: current_term: 1 leader_uuid: "09933f6893dd454b94dbadfdf55768ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } health_report { overall_health: UNKNOWN } } }
W20260620 23:22:23.837343 8188 tablet.cc:2369] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:22:23.854894 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:23.850439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.0.97.1:40741)
0620 23:22:23.850584 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:23.850588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:23.850603 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:22:23.850777 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:23.850781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:23.850855 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:23.850974 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:22:23.851266 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.851976 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.853133 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:23.853512 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:23.853597 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:23.853920 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:22:23.854392 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:22:23.854500 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:23.854569 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 23:22:23.854623 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:22:23.862648 8276 raft_consensus.cc:1261] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Refusing update from remote peer 09933f6893dd454b94dbadfdf55768ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:23.862648 8420 raft_consensus.cc:1261] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Refusing update from remote peer 09933f6893dd454b94dbadfdf55768ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:23.863695 8500 consensus_queue.cc:1035] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:23.864054 8495 consensus_queue.cc:1035] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:23.886657 7980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:23.886972 7980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:23.890184 7980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26: Sending DeleteTablet for 3 replicas of tablet f8e0bcc304dd4ee294b9cda16d710320
I20260620 23:22:23.891165 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet f8e0bcc304dd4ee294b9cda16d710320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:23 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55258
I20260620 23:22:23.891350 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet f8e0bcc304dd4ee294b9cda16d710320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:23 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42952
I20260620 23:22:23.892239 8513 tablet_replica.cc:330] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed: stopping tablet replica
I20260620 23:22:23.892261 8256 tablet_service.cc:1514] Processing DeleteTablet for tablet f8e0bcc304dd4ee294b9cda16d710320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:23 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44974
I20260620 23:22:23.892524 7980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 23:22:23.892750 8513 raft_consensus.cc:2229] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 1 LEADER]: Raft consensus shutting down.
W20260620 23:22:23.893334 7980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:22:23.893522 8513 raft_consensus.cc:2258] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:23.895148 8514 tablet_replica.cc:330] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6: stopping tablet replica
I20260620 23:22:23.895532 8514 raft_consensus.cc:2229] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:23.895983 8514 raft_consensus.cc:2258] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:23.898008 8514 ts_tablet_manager.cc:1905] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:23.899792 8515 tablet_replica.cc:330] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888: stopping tablet replica
I20260620 23:22:23.901074 8515 raft_consensus.cc:2229] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:23.901273 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 919c247342ad4a6584e65abb9fc446ea (DEFAULT_TABLE table=test-table [id=7097f9170276451f96b68dba857dc904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:23.901613 8515 raft_consensus.cc:2258] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:23.901691 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919c247342ad4a6584e65abb9fc446ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:23.902043 8513 ts_tablet_manager.cc:1905] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:23.905223 8514 ts_tablet_manager.cc:1918] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:23.905375 8514 log.cc:1196] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/f8e0bcc304dd4ee294b9cda16d710320
I20260620 23:22:23.906474 8255 tablet_service.cc:1467] Processing CreateTablet for tablet 919c247342ad4a6584e65abb9fc446ea (DEFAULT_TABLE table=test-table [id=7097f9170276451f96b68dba857dc904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:23.906474 8117 tablet_service.cc:1467] Processing CreateTablet for tablet 919c247342ad4a6584e65abb9fc446ea (DEFAULT_TABLE table=test-table [id=7097f9170276451f96b68dba857dc904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:23.906893 8255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919c247342ad4a6584e65abb9fc446ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:23.906909 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919c247342ad4a6584e65abb9fc446ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:23.907115 8515 ts_tablet_manager.cc:1905] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:23.908277 8514 ts_tablet_manager.cc:1939] T f8e0bcc304dd4ee294b9cda16d710320 P eb43c4a372264ba7a02626c6feb31ca6: Deleting consensus metadata
I20260620 23:22:23.911484 7966 catalog_manager.cc:4841] TS eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951): tablet f8e0bcc304dd4ee294b9cda16d710320 (table test-table [id=84caa151d4db40d5a3cc8bfe5fe4b38d]) successfully deleted
I20260620 23:22:23.915023 8489 tablet_bootstrap.cc:492] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888: Bootstrap starting.
I20260620 23:22:23.916154 8515 ts_tablet_manager.cc:1918] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:23.916307 8515 log.cc:1196] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/f8e0bcc304dd4ee294b9cda16d710320
I20260620 23:22:23.916612 8489 tablet_bootstrap.cc:654] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:23.916863 8515 ts_tablet_manager.cc:1939] T f8e0bcc304dd4ee294b9cda16d710320 P 540297eeb08149119ac1aca54f874888: Deleting consensus metadata
I20260620 23:22:23.916896 8491 tablet_bootstrap.cc:492] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6: Bootstrap starting.
I20260620 23:22:23.917006 8490 tablet_bootstrap.cc:492] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed: Bootstrap starting.
I20260620 23:22:23.918390 8491 tablet_bootstrap.cc:654] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:23.918427 7964 catalog_manager.cc:4841] TS 540297eeb08149119ac1aca54f874888 (127.0.97.2:41001): tablet f8e0bcc304dd4ee294b9cda16d710320 (table test-table [id=84caa151d4db40d5a3cc8bfe5fe4b38d]) successfully deleted
I20260620 23:22:23.918576 8490 tablet_bootstrap.cc:654] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:23.921818 8491 tablet_bootstrap.cc:492] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6: No bootstrap required, opened a new log
I20260620 23:22:23.921973 8489 tablet_bootstrap.cc:492] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888: No bootstrap required, opened a new log
I20260620 23:22:23.922093 8489 ts_tablet_manager.cc:1397] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 23:22:23.922080 8491 ts_tablet_manager.cc:1397] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 23:22:23.922768 8513 ts_tablet_manager.cc:1918] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:23.922920 8513 log.cc:1196] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/f8e0bcc304dd4ee294b9cda16d710320
I20260620 23:22:23.922852 8489 raft_consensus.cc:348] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:23.923058 8489 raft_consensus.cc:374] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:23.923029 8491 raft_consensus.cc:348] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:23.923146 8489 raft_consensus.cc:729] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540297eeb08149119ac1aca54f874888, State: Initialized, Role: FOLLOWER
I20260620 23:22:23.923229 8491 raft_consensus.cc:374] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:23.923307 8491 raft_consensus.cc:729] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb43c4a372264ba7a02626c6feb31ca6, State: Initialized, Role: FOLLOWER
I20260620 23:22:23.923452 8513 ts_tablet_manager.cc:1939] T f8e0bcc304dd4ee294b9cda16d710320 P 09933f6893dd454b94dbadfdf55768ed: Deleting consensus metadata
I20260620 23:22:23.923589 8491 consensus_queue.cc:260] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:23.923691 8489 consensus_queue.cc:260] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:23.924316 8491 ts_tablet_manager.cc:1428] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:22:23.924863 8489 ts_tablet_manager.cc:1428] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:22:23.925194 7964 catalog_manager.cc:4841] TS 09933f6893dd454b94dbadfdf55768ed (127.0.97.1:40741): tablet f8e0bcc304dd4ee294b9cda16d710320 (table test-table [id=84caa151d4db40d5a3cc8bfe5fe4b38d]) successfully deleted
I20260620 23:22:23.932010 8490 tablet_bootstrap.cc:492] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed: No bootstrap required, opened a new log
I20260620 23:22:23.932202 8490 ts_tablet_manager.cc:1397] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20260620 23:22:23.932993 8490 raft_consensus.cc:348] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:23.933243 8490 raft_consensus.cc:374] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:23.933339 8490 raft_consensus.cc:729] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09933f6893dd454b94dbadfdf55768ed, State: Initialized, Role: FOLLOWER
I20260620 23:22:23.933604 8490 consensus_queue.cc:260] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:23.934185 8490 ts_tablet_manager.cc:1428] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 23:22:24.133316 8508 raft_consensus.cc:482] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:24.133594 8508 raft_consensus.cc:504] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.134207 8496 raft_consensus.cc:482] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:24.134496 8508 leader_election.cc:288] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 540297eeb08149119ac1aca54f874888 (127.0.97.2:41001), eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951)
I20260620 23:22:24.134433 8496 raft_consensus.cc:504] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.135044 8276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c247342ad4a6584e65abb9fc446ea" candidate_uuid: "09933f6893dd454b94dbadfdf55768ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540297eeb08149119ac1aca54f874888" is_pre_election: true
I20260620 23:22:24.135138 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c247342ad4a6584e65abb9fc446ea" candidate_uuid: "09933f6893dd454b94dbadfdf55768ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb43c4a372264ba7a02626c6feb31ca6" is_pre_election: true
I20260620 23:22:24.135274 8276 raft_consensus.cc:2454] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09933f6893dd454b94dbadfdf55768ed in term 0.
I20260620 23:22:24.135347 8420 raft_consensus.cc:2454] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09933f6893dd454b94dbadfdf55768ed in term 0.
I20260620 23:22:24.135772 8067 leader_election.cc:302] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09933f6893dd454b94dbadfdf55768ed, 540297eeb08149119ac1aca54f874888, eb43c4a372264ba7a02626c6feb31ca6; no voters:
I20260620 23:22:24.136176 8508 raft_consensus.cc:2781] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:22:24.136283 8508 raft_consensus.cc:482] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:22:24.136289 8496 leader_election.cc:288] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09933f6893dd454b94dbadfdf55768ed (127.0.97.1:40741), eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951)
I20260620 23:22:24.136372 8508 raft_consensus.cc:3037] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:24.140542 8508 raft_consensus.cc:504] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.141767 8508 leader_election.cc:288] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [CANDIDATE]: Term 1 election: Requested vote from peers 540297eeb08149119ac1aca54f874888 (127.0.97.2:41001), eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951)
I20260620 23:22:24.142076 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c247342ad4a6584e65abb9fc446ea" candidate_uuid: "09933f6893dd454b94dbadfdf55768ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb43c4a372264ba7a02626c6feb31ca6"
I20260620 23:22:24.142313 8420 raft_consensus.cc:3037] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:24.145287 8420 raft_consensus.cc:2454] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09933f6893dd454b94dbadfdf55768ed in term 1.
I20260620 23:22:24.145748 8069 leader_election.cc:302] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09933f6893dd454b94dbadfdf55768ed, eb43c4a372264ba7a02626c6feb31ca6; no voters:
I20260620 23:22:24.146049 8508 raft_consensus.cc:2781] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:24.146183 8508 raft_consensus.cc:686] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 1 LEADER]: Becoming Leader. State: Replica: 09933f6893dd454b94dbadfdf55768ed, State: Running, Role: LEADER
I20260620 23:22:24.146487 8508 consensus_queue.cc:237] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.149711 8276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c247342ad4a6584e65abb9fc446ea" candidate_uuid: "09933f6893dd454b94dbadfdf55768ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540297eeb08149119ac1aca54f874888"
I20260620 23:22:24.149953 8276 raft_consensus.cc:3037] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:24.151908 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.136959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35147 (local address 127.0.97.1:40741)
0620 23:22:24.137130 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:24.137136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:24.142439 (+ 5303us) server_negotiation.cc:407] Connection header received
0620 23:22:24.142723 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:24.142729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:24.142801 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:24.142875 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:24.143895 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.145391 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.148933 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.150512 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.150631 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.150766 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:24.151448 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 23:22:24.151593 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:22:24.152511 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.136597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.97.2:35147)
0620 23:22:24.142331 (+ 5734us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:24.142352 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:24.142548 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:24.143092 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:24.143100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:24.143137 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:24.143678 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.143693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.145638 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.145646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.148717 (+ 3071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.148737 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.150964 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.150973 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.151207 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.151247 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:22:24.152055 (+ 808us) client_negotiation.cc:240] Negotiation successful
0620 23:22:24.152192 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5606,"thread_start_us":13733,"threads_started":1}
I20260620 23:22:24.153167 8276 raft_consensus.cc:2454] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09933f6893dd454b94dbadfdf55768ed in term 1.
I20260620 23:22:24.154101 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c247342ad4a6584e65abb9fc446ea" candidate_uuid: "540297eeb08149119ac1aca54f874888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09933f6893dd454b94dbadfdf55768ed" is_pre_election: true
I20260620 23:22:24.156239 7980 catalog_manager.cc:5495] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 09933f6893dd454b94dbadfdf55768ed (127.0.97.1). New cstate: current_term: 1 leader_uuid: "09933f6893dd454b94dbadfdf55768ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } health_report { overall_health: UNKNOWN } } }
I20260620 23:22:24.157274 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.137203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57987 (local address 127.0.97.3:40951)
0620 23:22:24.137384 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:24.137391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:24.150323 (+ 12932us) server_negotiation.cc:407] Connection header received
0620 23:22:24.150732 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:24.150736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:24.150817 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:24.150876 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:24.151709 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.152717 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.155192 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.156268 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.156413 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.156512 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:24.156893 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 23:22:24.157027 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:22:24.157298 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.136890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40951 (local address 127.0.97.2:57987)
0620 23:22:24.150228 (+ 13338us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:24.150247 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:24.150565 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:24.151072 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:24.151077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:24.151105 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:24.151519 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.151530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.153026 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.153030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.155025 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.155037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.156514 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.156523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.156710 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.156759 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:22:24.156869 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:22:24.156991 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13202,"thread_start_us":11975,"threads_started":1}
I20260620 23:22:24.157785 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c247342ad4a6584e65abb9fc446ea" candidate_uuid: "540297eeb08149119ac1aca54f874888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb43c4a372264ba7a02626c6feb31ca6" is_pre_election: true
I20260620 23:22:24.158094 8420 raft_consensus.cc:2379] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 540297eeb08149119ac1aca54f874888 in current term 1: Already voted for candidate 09933f6893dd454b94dbadfdf55768ed in this term.
I20260620 23:22:24.158579 8209 leader_election.cc:302] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 540297eeb08149119ac1aca54f874888; no voters: 09933f6893dd454b94dbadfdf55768ed, eb43c4a372264ba7a02626c6feb31ca6
I20260620 23:22:24.159008 8496 raft_consensus.cc:2726] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Jun 20 23:22:24 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997741, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:24.234990 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.221018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.0.97.62:42187)
0620 23:22:24.221951 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:24.221954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:24.221971 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:22:24.222038 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:24.222043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:24.222104 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:24.222209 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:24.223255 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.224037 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.225275 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.225655 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.225789 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.229378 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:24.229383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:24.229386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:24.229413 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:24.232357 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:24.233002 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:24.233024 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:24.233027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:24.233160 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:24.233545 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:24.233552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:24.233555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:24.233794 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:24.233958 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:24.234332 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 23:22:24.234539 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":651,"threads_started":1}
I20260620 23:22:24.235065 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.220987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42187 (local address 127.0.0.1:43312)
0620 23:22:24.221606 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:24.221631 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:24.221809 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:24.222237 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:24.222243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:24.222913 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:24.223115 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.223124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.224179 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.224190 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.225144 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.225155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.225812 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.225817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.225986 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.226736 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:24.226771 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:24.229177 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:24.232483 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:24.232495 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:24.232503 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:24.232889 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:24.233264 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:24.233271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:24.233274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:24.233429 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:24.233925 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:24.233935 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:24.234097 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 23:22:24.234260 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 23:22:24.234666 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":120,"thread_start_us":350,"threads_started":1}
I20260620 23:22:24.240231 7980 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43312:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:22:24.241178 7980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:22:24.248862 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 564b5e77df6c496c80d36d191100984a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a008fd0defde4578b55e48320ed86bc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:22:24.249053 8117 tablet_service.cc:1467] Processing CreateTablet for tablet 564b5e77df6c496c80d36d191100984a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a008fd0defde4578b55e48320ed86bc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:22:24.249181 8255 tablet_service.cc:1467] Processing CreateTablet for tablet 564b5e77df6c496c80d36d191100984a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a008fd0defde4578b55e48320ed86bc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:22:24.249444 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564b5e77df6c496c80d36d191100984a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:24.249604 8255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564b5e77df6c496c80d36d191100984a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:24.249599 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564b5e77df6c496c80d36d191100984a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:24.256480 8489 tablet_bootstrap.cc:492] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: Bootstrap starting.
I20260620 23:22:24.256564 8491 tablet_bootstrap.cc:492] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6: Bootstrap starting.
I20260620 23:22:24.257691 8489 tablet_bootstrap.cc:654] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:24.258117 8491 tablet_bootstrap.cc:654] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:24.259660 8489 tablet_bootstrap.cc:492] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: No bootstrap required, opened a new log
I20260620 23:22:24.259835 8489 ts_tablet_manager.cc:1397] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:22:24.260404 8489 raft_consensus.cc:348] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.260608 8489 raft_consensus.cc:374] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:24.260684 8489 raft_consensus.cc:729] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540297eeb08149119ac1aca54f874888, State: Initialized, Role: FOLLOWER
I20260620 23:22:24.260922 8489 consensus_queue.cc:260] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.261318 8496 tablet_replica.cc:439] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } }
I20260620 23:22:24.261560 8496 tablet_replica.cc:442] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:22:24.262092 8489 ts_tablet_manager.cc:1428] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:22:24.262802 8490 tablet_bootstrap.cc:492] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed: Bootstrap starting.
I20260620 23:22:24.264474 8490 tablet_bootstrap.cc:654] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:24.268663 8491 tablet_bootstrap.cc:492] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6: No bootstrap required, opened a new log
I20260620 23:22:24.268831 8491 ts_tablet_manager.cc:1397] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.006s
I20260620 23:22:24.269580 8491 raft_consensus.cc:348] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.269817 8491 raft_consensus.cc:374] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:24.269893 8491 raft_consensus.cc:729] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb43c4a372264ba7a02626c6feb31ca6, State: Initialized, Role: FOLLOWER
I20260620 23:22:24.270115 8491 consensus_queue.cc:260] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.270534 8497 tablet_replica.cc:439] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } }
I20260620 23:22:24.270834 8497 tablet_replica.cc:442] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:22:24.271622 8491 ts_tablet_manager.cc:1428] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 23:22:24.275156 8490 tablet_bootstrap.cc:492] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed: No bootstrap required, opened a new log
I20260620 23:22:24.275342 8490 ts_tablet_manager.cc:1397] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260620 23:22:24.276017 8490 raft_consensus.cc:348] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.276221 8490 raft_consensus.cc:374] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:24.276331 8490 raft_consensus.cc:729] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09933f6893dd454b94dbadfdf55768ed, State: Initialized, Role: FOLLOWER
I20260620 23:22:24.276558 8490 consensus_queue.cc:260] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.276970 8508 tablet_replica.cc:439] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } }
I20260620 23:22:24.277242 8508 tablet_replica.cc:442] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:22:24.277654 8490 ts_tablet_manager.cc:1428] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:22:24.310436 8496 raft_consensus.cc:482] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:24.310624 8496 raft_consensus.cc:504] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.311368 8496 leader_election.cc:288] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09933f6893dd454b94dbadfdf55768ed (127.0.97.1:40741), eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951)
I20260620 23:22:24.311817 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564b5e77df6c496c80d36d191100984a" candidate_uuid: "540297eeb08149119ac1aca54f874888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09933f6893dd454b94dbadfdf55768ed" is_pre_election: true
I20260620 23:22:24.311959 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564b5e77df6c496c80d36d191100984a" candidate_uuid: "540297eeb08149119ac1aca54f874888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb43c4a372264ba7a02626c6feb31ca6" is_pre_election: true
I20260620 23:22:24.312117 8142 raft_consensus.cc:2454] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 540297eeb08149119ac1aca54f874888 in term 0.
I20260620 23:22:24.312193 8420 raft_consensus.cc:2454] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 540297eeb08149119ac1aca54f874888 in term 0.
I20260620 23:22:24.312526 8207 leader_election.cc:302] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09933f6893dd454b94dbadfdf55768ed, 540297eeb08149119ac1aca54f874888; no voters:
I20260620 23:22:24.312952 8496 raft_consensus.cc:2781] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:22:24.313072 8496 raft_consensus.cc:482] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:22:24.313179 8496 raft_consensus.cc:3037] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:24.315479 8496 raft_consensus.cc:504] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.316030 8496 leader_election.cc:288] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [CANDIDATE]: Term 1 election: Requested vote from peers 09933f6893dd454b94dbadfdf55768ed (127.0.97.1:40741), eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951)
I20260620 23:22:24.316376 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564b5e77df6c496c80d36d191100984a" candidate_uuid: "540297eeb08149119ac1aca54f874888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09933f6893dd454b94dbadfdf55768ed"
I20260620 23:22:24.316457 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564b5e77df6c496c80d36d191100984a" candidate_uuid: "540297eeb08149119ac1aca54f874888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb43c4a372264ba7a02626c6feb31ca6"
I20260620 23:22:24.316569 8142 raft_consensus.cc:3037] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:24.316684 8420 raft_consensus.cc:3037] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:24.318795 8142 raft_consensus.cc:2454] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 540297eeb08149119ac1aca54f874888 in term 1.
I20260620 23:22:24.319442 8207 leader_election.cc:302] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09933f6893dd454b94dbadfdf55768ed, 540297eeb08149119ac1aca54f874888; no voters:
I20260620 23:22:24.319608 8420 raft_consensus.cc:2454] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 540297eeb08149119ac1aca54f874888 in term 1.
I20260620 23:22:24.319717 8496 raft_consensus.cc:2781] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:24.320210 8496 raft_consensus.cc:686] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [term 1 LEADER]: Becoming Leader. State: Replica: 540297eeb08149119ac1aca54f874888, State: Running, Role: LEADER
I20260620 23:22:24.320623 8496 consensus_queue.cc:237] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.323319 8532 tablet_replica.cc:439] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: TxnStatusTablet state changed. Reason: New leader 540297eeb08149119ac1aca54f874888. Latest consensus state: current_term: 1 leader_uuid: "540297eeb08149119ac1aca54f874888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } }
I20260620 23:22:24.323526 8532 tablet_replica.cc:442] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:22:24.324108 8534 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:22:24.323894 7980 catalog_manager.cc:5495] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 540297eeb08149119ac1aca54f874888 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "540297eeb08149119ac1aca54f874888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } health_report { overall_health: UNKNOWN } } }
I20260620 23:22:24.324298 8534 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:22:24.352427 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.345843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.0.97.2:41001)
0620 23:22:24.346434 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:24.346440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:24.346456 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:22:24.346522 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:24.346525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:24.346589 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:24.346795 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:22:24.347217 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.348035 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.349920 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.350603 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.350748 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.351308 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:22:24.351841 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:22:24.351946 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:24.352100 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:22:24.352160 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":317,"threads_started":1}
I20260620 23:22:24.352525 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.345730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41001 (local address 127.0.0.1:44982)
0620 23:22:24.345923 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:24.345941 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:24.346087 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:24.346809 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:24.346813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:24.346824 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:22:24.347068 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.347077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.348175 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.348183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.349788 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.349803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.350784 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.350791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.351022 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.351091 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:22:24.351948 (+ 857us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:22:24.351973 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 23:22:24.352061 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:22:24.352210 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:22:24.356237 8420 raft_consensus.cc:1261] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Refusing update from remote peer 540297eeb08149119ac1aca54f874888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:24.356467 8142 raft_consensus.cc:1261] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Refusing update from remote peer 540297eeb08149119ac1aca54f874888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:24.356918 8532 consensus_queue.cc:1035] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:24.357218 8496 consensus_queue.cc:1035] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:24.360626 8497 tablet_replica.cc:439] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6: TxnStatusTablet state changed. Reason: New leader 540297eeb08149119ac1aca54f874888. Latest consensus state: current_term: 1 leader_uuid: "540297eeb08149119ac1aca54f874888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } }
I20260620 23:22:24.360954 8497 tablet_replica.cc:442] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:22:24.363355 8496 tablet_replica.cc:439] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "540297eeb08149119ac1aca54f874888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } }
I20260620 23:22:24.363776 8496 tablet_replica.cc:442] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:22:24.364024 8505 mvcc.cc:205] Tried to move back new op lower bound from 7299062760876285952 to 7299062760740671488. Current Snapshot: MvccSnapshot[applied={T|T < 7299062760876285952 or (T in {7299062760876285952})}]
I20260620 23:22:24.367090 8504 mvcc.cc:205] Tried to move back new op lower bound from 7299062760876285952 to 7299062760740671488. Current Snapshot: MvccSnapshot[applied={T|T < 7299062760876285952 or (T in {7299062760876285952})}]
I20260620 23:22:24.367682 8497 tablet_replica.cc:439] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "540297eeb08149119ac1aca54f874888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } }
I20260620 23:22:24.367941 8497 tablet_replica.cc:442] T 564b5e77df6c496c80d36d191100984a P eb43c4a372264ba7a02626c6feb31ca6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:22:24.372339 8508 tablet_replica.cc:439] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed: TxnStatusTablet state changed. Reason: New leader 540297eeb08149119ac1aca54f874888. Latest consensus state: current_term: 1 leader_uuid: "540297eeb08149119ac1aca54f874888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } }
I20260620 23:22:24.372561 8508 tablet_replica.cc:442] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:22:24.374549 8532 tablet_replica.cc:439] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "540297eeb08149119ac1aca54f874888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } }
I20260620 23:22:24.374821 8532 tablet_replica.cc:442] T 564b5e77df6c496c80d36d191100984a P 540297eeb08149119ac1aca54f874888: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:22:24.376014 8508 tablet_replica.cc:439] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "540297eeb08149119ac1aca54f874888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } } }
I20260620 23:22:24.376293 8508 tablet_replica.cc:442] T 564b5e77df6c496c80d36d191100984a P 09933f6893dd454b94dbadfdf55768ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:22:24.388055 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.378461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41001 (local address 127.0.97.1:49865)
0620 23:22:24.383025 (+ 4564us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:24.383045 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:24.383153 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:24.383471 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:24.383474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:24.383483 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:22:24.383689 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.383696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.385007 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.385010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.385989 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.385999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.387017 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.387021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.387163 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.387213 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:22:24.387600 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:22:24.387607 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:22:24.387674 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:22:24.387761 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4416,"thread_start_us":4294,"threads_started":1}
I20260620 23:22:24.388118 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.378698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49865 (local address 127.0.97.2:41001)
0620 23:22:24.378874 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:24.378880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:24.383129 (+ 4249us) server_negotiation.cc:407] Connection header received
0620 23:22:24.383266 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:24.383272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:24.383357 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:24.383458 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:22:24.383825 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.384885 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.386118 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.386865 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.387002 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.387375 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:22:24.387502 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:22:24.387595 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:24.387717 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 23:22:24.387807 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:22:24.397291 8420 raft_consensus.cc:1261] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Refusing update from remote peer 09933f6893dd454b94dbadfdf55768ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:24.397311 8276 raft_consensus.cc:1261] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Refusing update from remote peer 09933f6893dd454b94dbadfdf55768ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:24.397850 8495 consensus_queue.cc:1035] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:24.398072 8508 consensus_queue.cc:1035] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:24.402508 8503 mvcc.cc:205] Tried to move back new op lower bound from 7299062761047044096 to 7299062760026644480. Current Snapshot: MvccSnapshot[applied={T|T < 7299062761047044096}]
I20260620 23:22:24.441401 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.423318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54211 (local address 127.0.97.1:40741)
0620 23:22:24.423521 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:24.423529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:24.436430 (+ 12901us) server_negotiation.cc:407] Connection header received
0620 23:22:24.436626 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:24.436631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:24.436712 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:24.436789 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:22:24.437308 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.438263 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.439772 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.440143 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.440273 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.440654 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:22:24.440769 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:22:24.440872 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:24.441026 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:22:24.441093 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:22:24.441509 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.423264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.97.2:54211)
0620 23:22:24.436332 (+ 13068us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:24.436347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:24.436502 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:24.436811 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:24.436815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:24.436827 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:22:24.437135 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.437147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.438434 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.438439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.439631 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.439644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.440311 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.440318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.440442 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.440493 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:22:24.440881 (+ 388us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:22:24.440899 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 23:22:24.440996 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:22:24.441113 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12892,"thread_start_us":13333,"threads_started":1}
I20260620 23:22:24.471468 7979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:24.471692 7979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:24.474314 7979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26: Sending DeleteTablet for 3 replicas of tablet 919c247342ad4a6584e65abb9fc446ea
I20260620 23:22:24.474974 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet 919c247342ad4a6584e65abb9fc446ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55258
I20260620 23:22:24.475150 8255 tablet_service.cc:1514] Processing DeleteTablet for tablet 919c247342ad4a6584e65abb9fc446ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44974
I20260620 23:22:24.475253 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet 919c247342ad4a6584e65abb9fc446ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42952
I20260620 23:22:24.485319 8551 tablet_replica.cc:330] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888: stopping tablet replica
I20260620 23:22:24.485611 8551 raft_consensus.cc:2229] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:24.485961 8551 raft_consensus.cc:2258] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:24.487083 8551 ts_tablet_manager.cc:1905] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:24.488077 8550 tablet_replica.cc:330] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed: stopping tablet replica
I20260620 23:22:24.488704 8550 raft_consensus.cc:2229] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:22:24.489344 8550 raft_consensus.cc:2258] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:24.490988 8550 ts_tablet_manager.cc:1905] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:24.492702 8552 tablet_replica.cc:330] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6: stopping tablet replica
I20260620 23:22:24.492982 8552 raft_consensus.cc:2229] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:24.493386 8552 raft_consensus.cc:2258] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:24.494387 8551 ts_tablet_manager.cc:1918] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:22:24.494593 8551 log.cc:1196] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/919c247342ad4a6584e65abb9fc446ea
I20260620 23:22:24.495168 8551 ts_tablet_manager.cc:1939] T 919c247342ad4a6584e65abb9fc446ea P 540297eeb08149119ac1aca54f874888: Deleting consensus metadata
I20260620 23:22:24.495249 8552 ts_tablet_manager.cc:1905] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:24.496235 7964 catalog_manager.cc:4841] TS 540297eeb08149119ac1aca54f874888 (127.0.97.2:41001): tablet 919c247342ad4a6584e65abb9fc446ea (table test-table [id=7097f9170276451f96b68dba857dc904]) successfully deleted
I20260620 23:22:24.501519 8550 ts_tablet_manager.cc:1918] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:22:24.501709 8550 log.cc:1196] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/919c247342ad4a6584e65abb9fc446ea
I20260620 23:22:24.502172 8550 ts_tablet_manager.cc:1939] T 919c247342ad4a6584e65abb9fc446ea P 09933f6893dd454b94dbadfdf55768ed: Deleting consensus metadata
I20260620 23:22:24.503186 7964 catalog_manager.cc:4841] TS 09933f6893dd454b94dbadfdf55768ed (127.0.97.1:40741): tablet 919c247342ad4a6584e65abb9fc446ea (table test-table [id=7097f9170276451f96b68dba857dc904]) successfully deleted
I20260620 23:22:24.503352 8552 ts_tablet_manager.cc:1918] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:22:24.503516 8552 log.cc:1196] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/919c247342ad4a6584e65abb9fc446ea
I20260620 23:22:24.503960 8552 ts_tablet_manager.cc:1939] T 919c247342ad4a6584e65abb9fc446ea P eb43c4a372264ba7a02626c6feb31ca6: Deleting consensus metadata
I20260620 23:22:24.505014 7966 catalog_manager.cc:4841] TS eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951): tablet 919c247342ad4a6584e65abb9fc446ea (table test-table [id=7097f9170276451f96b68dba857dc904]) successfully deleted
Jun 20 23:22:24 dist-test-slave-5kzz krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:22:24 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:24 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:22:24.543040 8528 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:24.528610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.0.97.62:42187)
0620 23:22:24.528843 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:24.528853 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:24.529333 (+ 480us) server_negotiation.cc:407] Connection header received
0620 23:22:24.529510 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:24.529514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:24.529641 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:24.529724 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:24.531329 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.532118 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.533872 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.534265 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.534355 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.542820 (+ 8465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43318: BlockingRecv error: recv got EOF from 127.0.0.1:43318 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 23:22:24 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:22:24.560909 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.545999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.0.97.62:42187)
0620 23:22:24.546216 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:24.546222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:24.546491 (+ 269us) server_negotiation.cc:407] Connection header received
0620 23:22:24.546627 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:24.546630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:24.546727 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:24.546824 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:24.548198 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.549040 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.550783 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.551601 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.551713 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.555799 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:24.555804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:24.555807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:24.555836 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:24.558327 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:24.559021 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:24.559027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:24.559031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:24.559148 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:24.559501 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:24.559507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:24.559510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:24.559760 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:24.560021 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:24.560322 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 23:22:24.560546 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:22:24.564262 7979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:22:24.564857 7979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:22:24.571166 8117 tablet_service.cc:1467] Processing CreateTablet for tablet fe4829e82bb04f66a1b5b4083760dfa9 (DEFAULT_TABLE table=test-table [id=3fb4be0430ca45c2b3f59a75587bb43c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:24.571267 8255 tablet_service.cc:1467] Processing CreateTablet for tablet fe4829e82bb04f66a1b5b4083760dfa9 (DEFAULT_TABLE table=test-table [id=3fb4be0430ca45c2b3f59a75587bb43c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:24.571619 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4829e82bb04f66a1b5b4083760dfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:24.571695 8255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4829e82bb04f66a1b5b4083760dfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:24.578107 8489 tablet_bootstrap.cc:492] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888: Bootstrap starting.
I20260620 23:22:24.579360 8489 tablet_bootstrap.cc:654] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:24.581306 8490 tablet_bootstrap.cc:492] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed: Bootstrap starting.
I20260620 23:22:24.581252 8395 tablet_service.cc:1467] Processing CreateTablet for tablet fe4829e82bb04f66a1b5b4083760dfa9 (DEFAULT_TABLE table=test-table [id=3fb4be0430ca45c2b3f59a75587bb43c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:24.581655 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4829e82bb04f66a1b5b4083760dfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:24.582479 8490 tablet_bootstrap.cc:654] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:24.588500 8489 tablet_bootstrap.cc:492] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888: No bootstrap required, opened a new log
I20260620 23:22:24.589326 8489 ts_tablet_manager.cc:1397] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20260620 23:22:24.590361 8489 raft_consensus.cc:348] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.590739 8489 raft_consensus.cc:374] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:24.590857 8489 raft_consensus.cc:729] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540297eeb08149119ac1aca54f874888, State: Initialized, Role: FOLLOWER
I20260620 23:22:24.591373 8489 consensus_queue.cc:260] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.593421 8489 ts_tablet_manager.cc:1428] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 23:22:24.594376 8491 tablet_bootstrap.cc:492] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6: Bootstrap starting.
I20260620 23:22:24.595844 8491 tablet_bootstrap.cc:654] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:24.606525 8491 tablet_bootstrap.cc:492] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6: No bootstrap required, opened a new log
I20260620 23:22:24.606707 8491 ts_tablet_manager.cc:1397] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260620 23:22:24.607630 8491 raft_consensus.cc:348] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.607992 8491 raft_consensus.cc:374] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:24.608144 8491 raft_consensus.cc:729] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb43c4a372264ba7a02626c6feb31ca6, State: Initialized, Role: FOLLOWER
I20260620 23:22:24.608433 8491 consensus_queue.cc:260] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.609375 8490 tablet_bootstrap.cc:492] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed: No bootstrap required, opened a new log
I20260620 23:22:24.609918 8491 ts_tablet_manager.cc:1428] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 23:22:24.610265 8490 ts_tablet_manager.cc:1397] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.015s
I20260620 23:22:24.611167 8490 raft_consensus.cc:348] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.611483 8490 raft_consensus.cc:374] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:24.611621 8490 raft_consensus.cc:729] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09933f6893dd454b94dbadfdf55768ed, State: Initialized, Role: FOLLOWER
I20260620 23:22:24.611907 8490 consensus_queue.cc:260] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.613698 8490 ts_tablet_manager.cc:1428] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:22:24.636783 8497 raft_consensus.cc:482] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:24.637022 8497 raft_consensus.cc:504] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.638617 8549 raft_consensus.cc:482] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:24.638862 8497 leader_election.cc:288] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09933f6893dd454b94dbadfdf55768ed (127.0.97.1:40741), 540297eeb08149119ac1aca54f874888 (127.0.97.2:41001)
I20260620 23:22:24.638839 8549 raft_consensus.cc:504] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.639572 8549 leader_election.cc:288] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09933f6893dd454b94dbadfdf55768ed (127.0.97.1:40741), eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951)
I20260620 23:22:24.640065 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4829e82bb04f66a1b5b4083760dfa9" candidate_uuid: "540297eeb08149119ac1aca54f874888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09933f6893dd454b94dbadfdf55768ed" is_pre_election: true
I20260620 23:22:24.640337 8141 raft_consensus.cc:2454] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 540297eeb08149119ac1aca54f874888 in term 0.
I20260620 23:22:24.640842 8207 leader_election.cc:302] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09933f6893dd454b94dbadfdf55768ed, 540297eeb08149119ac1aca54f874888; no voters:
I20260620 23:22:24.641162 8549 raft_consensus.cc:2781] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:22:24.641274 8549 raft_consensus.cc:482] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:22:24.641361 8549 raft_consensus.cc:3037] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:24.644495 8549 raft_consensus.cc:504] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.645560 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4829e82bb04f66a1b5b4083760dfa9" candidate_uuid: "540297eeb08149119ac1aca54f874888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09933f6893dd454b94dbadfdf55768ed"
I20260620 23:22:24.645747 8141 raft_consensus.cc:3037] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:24.649004 8141 raft_consensus.cc:2454] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 540297eeb08149119ac1aca54f874888 in term 1.
I20260620 23:22:24.649488 8207 leader_election.cc:302] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09933f6893dd454b94dbadfdf55768ed, 540297eeb08149119ac1aca54f874888; no voters:
I20260620 23:22:24.649772 8532 raft_consensus.cc:2781] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:24.649916 8532 raft_consensus.cc:686] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 1 LEADER]: Becoming Leader. State: Replica: 540297eeb08149119ac1aca54f874888, State: Running, Role: LEADER
I20260620 23:22:24.649936 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.639619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42615 (local address 127.0.97.1:40741)
0620 23:22:24.639912 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:24.639918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:24.640146 (+ 228us) server_negotiation.cc:407] Connection header received
0620 23:22:24.640380 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:24.640389 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:24.640464 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:24.640524 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:24.641262 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.642719 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.646044 (+ 3325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.647904 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.648036 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.648172 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:24.649377 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0620 23:22:24.649564 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 23:22:24.650185 8532 consensus_queue.cc:237] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.650761 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.639340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.97.3:42615)
0620 23:22:24.640047 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:24.640061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:24.640206 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:24.640721 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:24.640725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:24.640758 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:24.641061 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.641072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.642932 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.642940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.645817 (+ 2877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.645838 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.648852 (+ 3014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.648861 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.649138 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.649189 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:22:24.650111 (+ 922us) client_negotiation.cc:240] Negotiation successful
0620 23:22:24.650323 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":826,"threads_started":1}
I20260620 23:22:24.651533 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4829e82bb04f66a1b5b4083760dfa9" candidate_uuid: "540297eeb08149119ac1aca54f874888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb43c4a372264ba7a02626c6feb31ca6" is_pre_election: true
I20260620 23:22:24.651739 8420 raft_consensus.cc:2454] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 540297eeb08149119ac1aca54f874888 in term 0.
I20260620 23:22:24.652582 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4829e82bb04f66a1b5b4083760dfa9" candidate_uuid: "eb43c4a372264ba7a02626c6feb31ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09933f6893dd454b94dbadfdf55768ed" is_pre_election: true
I20260620 23:22:24.652863 8142 raft_consensus.cc:2379] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb43c4a372264ba7a02626c6feb31ca6 in current term 1: Already voted for candidate 540297eeb08149119ac1aca54f874888 in this term.
I20260620 23:22:24.653946 8549 leader_election.cc:288] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [CANDIDATE]: Term 1 election: Requested vote from peers 09933f6893dd454b94dbadfdf55768ed (127.0.97.1:40741), eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951)
I20260620 23:22:24.654353 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4829e82bb04f66a1b5b4083760dfa9" candidate_uuid: "540297eeb08149119ac1aca54f874888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb43c4a372264ba7a02626c6feb31ca6"
I20260620 23:22:24.654523 8420 raft_consensus.cc:3037] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:24.655020 7980 catalog_manager.cc:5495] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 540297eeb08149119ac1aca54f874888 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "540297eeb08149119ac1aca54f874888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } health_report { overall_health: UNKNOWN } } }
I20260620 23:22:24.657439 8420 raft_consensus.cc:2454] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 540297eeb08149119ac1aca54f874888 in term 1.
I20260620 23:22:24.659168 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.640516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41001 (local address 127.0.97.3:38615)
0620 23:22:24.652269 (+ 11753us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:24.652302 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:24.652493 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:24.653026 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:24.653030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:24.653055 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:24.653217 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.653227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.654877 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.654882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.656644 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:24.656657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.658401 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.658406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:24.658587 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.658617 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:22:24.658740 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 23:22:24.658886 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11644,"thread_start_us":10536,"threads_started":1}
I20260620 23:22:24.659188 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.640806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38615 (local address 127.0.97.2:41001)
0620 23:22:24.640991 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:24.640998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:24.652388 (+ 11390us) server_negotiation.cc:407] Connection header received
0620 23:22:24.652671 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:24.652680 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:24.652760 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:24.652829 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:24.653392 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.654665 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.656801 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.658084 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.658170 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.658269 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:24.658760 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 23:22:24.658888 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:22:24.659761 8276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4829e82bb04f66a1b5b4083760dfa9" candidate_uuid: "eb43c4a372264ba7a02626c6feb31ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540297eeb08149119ac1aca54f874888" is_pre_election: true
I20260620 23:22:24.660269 8346 leader_election.cc:302] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb43c4a372264ba7a02626c6feb31ca6; no voters: 09933f6893dd454b94dbadfdf55768ed, 540297eeb08149119ac1aca54f874888
I20260620 23:22:24.660653 8497 raft_consensus.cc:2726] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 23:22:24.677623 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:24.672364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.0.97.2:41001)
0620 23:22:24.672515 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:24.672522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:24.672542 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:22:24.672748 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:24.672751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:24.672809 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:24.672881 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:22:24.673276 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.674022 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.675982 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:24.676400 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:24.676484 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:24.676946 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:22:24.677071 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:22:24.677132 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:24.677300 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 23:22:24.677354 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:22:24.680912 8141 raft_consensus.cc:1261] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Refusing update from remote peer 540297eeb08149119ac1aca54f874888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:24.680912 8420 raft_consensus.cc:1261] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Refusing update from remote peer 540297eeb08149119ac1aca54f874888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:24.681438 8549 consensus_queue.cc:1035] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:24.681720 8532 consensus_queue.cc:1035] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:24.699957 7980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:24.700141 7980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:24.702967 7980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26: Sending DeleteTablet for 3 replicas of tablet fe4829e82bb04f66a1b5b4083760dfa9
I20260620 23:22:24.703718 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4829e82bb04f66a1b5b4083760dfa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55258
I20260620 23:22:24.703745 8255 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4829e82bb04f66a1b5b4083760dfa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44974
I20260620 23:22:24.704015 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4829e82bb04f66a1b5b4083760dfa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42952
I20260620 23:22:24.704264 8552 tablet_replica.cc:330] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6: stopping tablet replica
I20260620 23:22:24.704474 8552 raft_consensus.cc:2229] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:24.704684 8552 raft_consensus.cc:2258] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:24.704965 8551 tablet_replica.cc:330] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888: stopping tablet replica
I20260620 23:22:24.705180 8551 raft_consensus.cc:2229] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:22:24.705442 7980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 23:22:24.705595 8551 raft_consensus.cc:2258] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:24.705858 8550 tablet_replica.cc:330] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed: stopping tablet replica
W20260620 23:22:24.705989 7980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:22:24.706045 8550 raft_consensus.cc:2229] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:24.706251 8550 raft_consensus.cc:2258] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:24.707651 8552 ts_tablet_manager.cc:1905] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:24.707718 8551 ts_tablet_manager.cc:1905] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:24.713090 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 4b8a70194aea46a2a1e8bc373b53fe65 (DEFAULT_TABLE table=test-table [id=b0d79184a80945178cc666c0520565da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:24.713279 8255 tablet_service.cc:1467] Processing CreateTablet for tablet 4b8a70194aea46a2a1e8bc373b53fe65 (DEFAULT_TABLE table=test-table [id=b0d79184a80945178cc666c0520565da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:24.713253 8117 tablet_service.cc:1467] Processing CreateTablet for tablet 4b8a70194aea46a2a1e8bc373b53fe65 (DEFAULT_TABLE table=test-table [id=b0d79184a80945178cc666c0520565da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:24.713491 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b8a70194aea46a2a1e8bc373b53fe65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:24.713676 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b8a70194aea46a2a1e8bc373b53fe65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:24.713673 8255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b8a70194aea46a2a1e8bc373b53fe65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:24.719952 8490 tablet_bootstrap.cc:492] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed: Bootstrap starting.
I20260620 23:22:24.721042 8489 tablet_bootstrap.cc:492] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888: Bootstrap starting.
I20260620 23:22:24.721602 8490 tablet_bootstrap.cc:654] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:24.722417 8489 tablet_bootstrap.cc:654] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:24.723636 8490 tablet_bootstrap.cc:492] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed: No bootstrap required, opened a new log
I20260620 23:22:24.723790 8490 ts_tablet_manager.cc:1397] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260620 23:22:24.724381 8489 tablet_bootstrap.cc:492] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888: No bootstrap required, opened a new log
I20260620 23:22:24.724494 8489 ts_tablet_manager.cc:1397] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260620 23:22:24.724531 8490 raft_consensus.cc:348] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.724758 8490 raft_consensus.cc:374] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:24.724833 8490 raft_consensus.cc:729] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09933f6893dd454b94dbadfdf55768ed, State: Initialized, Role: FOLLOWER
I20260620 23:22:24.725096 8490 consensus_queue.cc:260] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.725931 8490 ts_tablet_manager.cc:1428] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 23:22:24.725396 8489 raft_consensus.cc:348] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.726320 8489 raft_consensus.cc:374] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:24.726475 8489 raft_consensus.cc:729] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540297eeb08149119ac1aca54f874888, State: Initialized, Role: FOLLOWER
I20260620 23:22:24.726897 8491 tablet_bootstrap.cc:492] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6: Bootstrap starting.
I20260620 23:22:24.726794 8489 consensus_queue.cc:260] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.727314 8489 ts_tablet_manager.cc:1428] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 23:22:24.728471 8491 tablet_bootstrap.cc:654] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:24.730150 8551 ts_tablet_manager.cc:1918] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:24.730386 8551 log.cc:1196] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/fe4829e82bb04f66a1b5b4083760dfa9
I20260620 23:22:24.731014 8551 ts_tablet_manager.cc:1939] T fe4829e82bb04f66a1b5b4083760dfa9 P 540297eeb08149119ac1aca54f874888: Deleting consensus metadata
I20260620 23:22:24.732348 8550 ts_tablet_manager.cc:1905] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:24.733458 8552 ts_tablet_manager.cc:1918] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:24.733644 8552 log.cc:1196] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/fe4829e82bb04f66a1b5b4083760dfa9
I20260620 23:22:24.734076 8552 ts_tablet_manager.cc:1939] T fe4829e82bb04f66a1b5b4083760dfa9 P eb43c4a372264ba7a02626c6feb31ca6: Deleting consensus metadata
I20260620 23:22:24.735123 7966 catalog_manager.cc:4841] TS eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951): tablet fe4829e82bb04f66a1b5b4083760dfa9 (table test-table [id=3fb4be0430ca45c2b3f59a75587bb43c]) successfully deleted
I20260620 23:22:24.737464 8491 tablet_bootstrap.cc:492] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6: No bootstrap required, opened a new log
I20260620 23:22:24.737613 7964 catalog_manager.cc:4841] TS 540297eeb08149119ac1aca54f874888 (127.0.97.2:41001): tablet fe4829e82bb04f66a1b5b4083760dfa9 (table test-table [id=3fb4be0430ca45c2b3f59a75587bb43c]) successfully deleted
I20260620 23:22:24.737664 8491 ts_tablet_manager.cc:1397] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20260620 23:22:24.738289 8550 ts_tablet_manager.cc:1918] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:24.738435 8550 log.cc:1196] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/fe4829e82bb04f66a1b5b4083760dfa9
I20260620 23:22:24.738432 8491 raft_consensus.cc:348] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.738705 8491 raft_consensus.cc:374] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:24.738791 8491 raft_consensus.cc:729] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb43c4a372264ba7a02626c6feb31ca6, State: Initialized, Role: FOLLOWER
I20260620 23:22:24.738893 8550 ts_tablet_manager.cc:1939] T fe4829e82bb04f66a1b5b4083760dfa9 P 09933f6893dd454b94dbadfdf55768ed: Deleting consensus metadata
I20260620 23:22:24.739037 8491 consensus_queue.cc:260] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:24.739509 8491 ts_tablet_manager.cc:1428] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:22:24.739826 7964 catalog_manager.cc:4841] TS 09933f6893dd454b94dbadfdf55768ed (127.0.97.1:40741): tablet fe4829e82bb04f66a1b5b4083760dfa9 (table test-table [id=3fb4be0430ca45c2b3f59a75587bb43c]) successfully deleted
I20260620 23:22:25.085512 8545 raft_consensus.cc:482] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:25.085737 8545 raft_consensus.cc:504] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:25.086498 8545 leader_election.cc:288] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 540297eeb08149119ac1aca54f874888 (127.0.97.2:41001), eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951)
I20260620 23:22:25.087009 8276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8a70194aea46a2a1e8bc373b53fe65" candidate_uuid: "09933f6893dd454b94dbadfdf55768ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540297eeb08149119ac1aca54f874888" is_pre_election: true
I20260620 23:22:25.087250 8276 raft_consensus.cc:2454] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09933f6893dd454b94dbadfdf55768ed in term 0.
I20260620 23:22:25.087312 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8a70194aea46a2a1e8bc373b53fe65" candidate_uuid: "09933f6893dd454b94dbadfdf55768ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb43c4a372264ba7a02626c6feb31ca6" is_pre_election: true
I20260620 23:22:25.087577 8420 raft_consensus.cc:2454] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09933f6893dd454b94dbadfdf55768ed in term 0.
I20260620 23:22:25.087731 8067 leader_election.cc:302] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09933f6893dd454b94dbadfdf55768ed, 540297eeb08149119ac1aca54f874888; no voters:
I20260620 23:22:25.088022 8545 raft_consensus.cc:2781] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:22:25.088135 8545 raft_consensus.cc:482] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:22:25.088214 8545 raft_consensus.cc:3037] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:25.091041 8545 raft_consensus.cc:504] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:25.091629 8545 leader_election.cc:288] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [CANDIDATE]: Term 1 election: Requested vote from peers 540297eeb08149119ac1aca54f874888 (127.0.97.2:41001), eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951)
I20260620 23:22:25.092073 8276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8a70194aea46a2a1e8bc373b53fe65" candidate_uuid: "09933f6893dd454b94dbadfdf55768ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540297eeb08149119ac1aca54f874888"
I20260620 23:22:25.092108 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8a70194aea46a2a1e8bc373b53fe65" candidate_uuid: "09933f6893dd454b94dbadfdf55768ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb43c4a372264ba7a02626c6feb31ca6"
I20260620 23:22:25.092315 8276 raft_consensus.cc:3037] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:25.092322 8420 raft_consensus.cc:3037] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:25.094818 8276 raft_consensus.cc:2454] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09933f6893dd454b94dbadfdf55768ed in term 1.
I20260620 23:22:25.095180 8067 leader_election.cc:302] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09933f6893dd454b94dbadfdf55768ed, 540297eeb08149119ac1aca54f874888; no voters:
I20260620 23:22:25.095376 8420 raft_consensus.cc:2454] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09933f6893dd454b94dbadfdf55768ed in term 1.
I20260620 23:22:25.095475 8545 raft_consensus.cc:2781] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:25.095960 8545 raft_consensus.cc:686] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 1 LEADER]: Becoming Leader. State: Replica: 09933f6893dd454b94dbadfdf55768ed, State: Running, Role: LEADER
I20260620 23:22:25.096271 8545 consensus_queue.cc:237] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } }
I20260620 23:22:25.098933 7978 catalog_manager.cc:5495] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 09933f6893dd454b94dbadfdf55768ed (127.0.97.1). New cstate: current_term: 1 leader_uuid: "09933f6893dd454b94dbadfdf55768ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09933f6893dd454b94dbadfdf55768ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 } health_report { overall_health: UNKNOWN } } }
I20260620 23:22:25.195274 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:25.190663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.0.97.1:40741)
0620 23:22:25.191122 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:25.191126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:25.191162 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:22:25.191307 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:25.191310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:25.191366 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:25.191450 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:22:25.191788 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:25.192467 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:25.193851 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:25.194264 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:25.194378 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:25.194737 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:22:25.194846 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:22:25.194908 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:25.195021 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 23:22:25.195071 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":235,"threads_started":1}
I20260620 23:22:25.202831 8420 raft_consensus.cc:1261] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Refusing update from remote peer 09933f6893dd454b94dbadfdf55768ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:25.202970 8276 raft_consensus.cc:1261] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Refusing update from remote peer 09933f6893dd454b94dbadfdf55768ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:25.203353 8577 consensus_queue.cc:1035] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb43c4a372264ba7a02626c6feb31ca6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:25.203682 8545 consensus_queue.cc:1035] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "540297eeb08149119ac1aca54f874888" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:25.244429 7978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:25.244671 7978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:25.247226 7978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85e41f399c0845f29a40ddb469796c26: Sending DeleteTablet for 3 replicas of tablet 4b8a70194aea46a2a1e8bc373b53fe65
I20260620 23:22:25.247876 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b8a70194aea46a2a1e8bc373b53fe65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55258
I20260620 23:22:25.247887 8255 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b8a70194aea46a2a1e8bc373b53fe65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44974
I20260620 23:22:25.247875 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b8a70194aea46a2a1e8bc373b53fe65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42952
I20260620 23:22:25.248972 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 8051
I20260620 23:22:25.251969 8590 tablet_replica.cc:330] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6: stopping tablet replica
I20260620 23:22:25.252162 8590 raft_consensus.cc:2229] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:25.252182 8588 tablet_replica.cc:330] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed: stopping tablet replica
I20260620 23:22:25.252379 8590 raft_consensus.cc:2258] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:25.252380 8588 raft_consensus.cc:2229] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:22:25.252748 8588 raft_consensus.cc:2258] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:25.253371 8590 ts_tablet_manager.cc:1905] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:25.253849 8588 ts_tablet_manager.cc:1905] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:25.259881 8590 ts_tablet_manager.cc:1918] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:22:25.260025 8590 log.cc:1196] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/4b8a70194aea46a2a1e8bc373b53fe65
I20260620 23:22:25.260453 8590 ts_tablet_manager.cc:1939] T 4b8a70194aea46a2a1e8bc373b53fe65 P eb43c4a372264ba7a02626c6feb31ca6: Deleting consensus metadata
Jun 20 23:22:25 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 23:22:25.261556 7966 catalog_manager.cc:4841] TS eb43c4a372264ba7a02626c6feb31ca6 (127.0.97.3:40951): tablet 4b8a70194aea46a2a1e8bc373b53fe65 (table test-table [id=b0d79184a80945178cc666c0520565da]) successfully deleted
I20260620 23:22:25.261662 8589 tablet_replica.cc:330] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888: stopping tablet replica
I20260620 23:22:25.261850 8589 raft_consensus.cc:2229] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:25.262120 8589 raft_consensus.cc:2258] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:25.262432 8588 ts_tablet_manager.cc:1918] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:22:25.262550 8588 log.cc:1196] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/4b8a70194aea46a2a1e8bc373b53fe65
I20260620 23:22:25.263019 8588 ts_tablet_manager.cc:1939] T 4b8a70194aea46a2a1e8bc373b53fe65 P 09933f6893dd454b94dbadfdf55768ed: Deleting consensus metadata
I20260620 23:22:25.263028 8589 ts_tablet_manager.cc:1905] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:25.264117 7964 catalog_manager.cc:4841] TS 09933f6893dd454b94dbadfdf55768ed (127.0.97.1:40741): tablet 4b8a70194aea46a2a1e8bc373b53fe65 (table test-table [id=b0d79184a80945178cc666c0520565da]) successfully deleted
I20260620 23:22:25.268121 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:25.250619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.0.97.1:40741)
0620 23:22:25.250800 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:25.250807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:25.251499 (+ 692us) server_negotiation.cc:407] Connection header received
0620 23:22:25.251690 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:25.251697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:25.251778 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:25.251868 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:25.253494 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:25.254529 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:25.256521 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:25.257130 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:25.257223 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:25.262279 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:25.262297 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:25.262317 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:25.262353 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:25.265622 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:25.266374 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:25.266381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:25.266383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:25.266509 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:25.266875 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:25.266878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:25.266880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:25.267177 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:25.267396 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:25.267706 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 23:22:25.267856 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:22:25.268344 8589 ts_tablet_manager.cc:1918] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:22:25.268476 8589 log.cc:1196] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/4b8a70194aea46a2a1e8bc373b53fe65
I20260620 23:22:25.268746 8182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:25.268939 8589 ts_tablet_manager.cc:1939] T 4b8a70194aea46a2a1e8bc373b53fe65 P 540297eeb08149119ac1aca54f874888: Deleting consensus metadata
I20260620 23:22:25.270202 7964 catalog_manager.cc:4841] TS 540297eeb08149119ac1aca54f874888 (127.0.97.2:41001): tablet 4b8a70194aea46a2a1e8bc373b53fe65 (table test-table [id=b0d79184a80945178cc666c0520565da]) successfully deleted
I20260620 23:22:25.386703 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 8051
I20260620 23:22:25.408345 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 8191
Jun 20 23:22:25 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 23:22:25.421830 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:25.409059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.0.97.2:41001)
0620 23:22:25.409501 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:25.409505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:25.409521 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:22:25.409573 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:25.409577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:25.409639 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:25.409727 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:25.410932 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:25.411657 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:25.412818 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:25.413178 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:25.413260 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:25.417012 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:25.417024 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:25.417038 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:25.417067 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:25.419770 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:25.420338 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:25.420344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:25.420347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:25.420472 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:25.420747 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:25.420750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:25.420752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:25.421020 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:25.421227 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:25.421406 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 23:22:25.421561 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
I20260620 23:22:25.422453 8321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:25.558532 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 8191
I20260620 23:22:25.588191 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 8330
Jun 20 23:22:25 dist-test-slave-5kzz krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 23:22:25.601781 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:25.588883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.0.97.3:40951)
0620 23:22:25.589355 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:25.589360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:25.589375 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:22:25.589427 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:25.589430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:25.589488 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:25.589576 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:25.590801 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:25.591514 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:25.592623 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:25.593007 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:25.593090 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:25.597035 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:25.597049 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:25.597064 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:25.597096 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:25.599683 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:25.600244 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:25.600250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:25.600252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:25.600386 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:25.600677 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:25.600680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:25.600682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:25.600996 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:25.601197 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:25.601395 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 23:22:25.601535 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":261,"threads_started":1}
I20260620 23:22:25.602430 8460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:25.695886 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 8330
I20260620 23:22:25.713213 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 7948
I20260620 23:22:25.714322 8020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:25.822455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 7948
2026-06-20T23:22:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5515 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:22:25 dist-test-slave-5kzz krb5kdc[8600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:22:25 dist-test-slave-5kzz krb5kdc[8600](info): set up 2 sockets
Jun 20 23:22:25 dist-test-slave-5kzz krb5kdc[8600](info): commencing operation
krb5kdc: starting...
W20260620 23:22:26.871649 388 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:22:26 dist-test-slave-5kzz krb5kdc[8600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:22:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:22:27.064244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36301
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:27.365257 8613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:27.365504 8613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:27.365656 8613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:27.372380 8613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:27.372463 8613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:27.372503 8613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:27.372546 8613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:27.372632 8613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:22:27.381546 8613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:27.382730 8613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:27.383945 8613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:27.397964 8621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:27.398003 8619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:27.398140 8618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:27.398775 8613 server_base.cc:1034] running on GCE node
I20260620 23:22:27.399488 8613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:27.400676 8613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:27.401860 8613 hybrid_clock.cc:648] HybridClock initialized: now 1781997747401827 us; error 32 us; skew 500 ppm
I20260620 23:22:27.402195 8613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T23:22:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1644 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:22:27 dist-test-slave-5kzz krb5kdc[8627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:22:27 dist-test-slave-5kzz krb5kdc[8627](info): set up 2 sockets
Jun 20 23:22:27 dist-test-slave-5kzz krb5kdc[8627](info): commencing operation
krb5kdc: starting...
W20260620 23:22:28.519376 388 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:22:28 dist-test-slave-5kzz krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:22:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:22:28.709533 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45519
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:29.008949 8640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:29.009262 8640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:29.009425 8640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:29.016201 8640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:29.016321 8640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:29.016424 8640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:29.016515 8640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:29.016597 8640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:22:29.025600 8640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:29.026829 8640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:29.028038 8640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:29.040539 8645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:29.040625 8646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:29.040704 8648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:29.041671 8640 server_base.cc:1034] running on GCE node
I20260620 23:22:29.042372 8640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:29.043794 8640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:29.045024 8640 hybrid_clock.cc:648] HybridClock initialized: now 1781997749044984 us; error 36 us; skew 500 ppm
I20260620 23:22:29.045360 8640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:29 dist-test-slave-5kzz krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:29.048522 8640 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:22:29.049913 8640 webserver.cc:460] Webserver started at http://127.0.97.62:35121/ using document root <none> and password file <none>
I20260620 23:22:29.050536 8640 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:29.050714 8640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:29.051023 8640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:29.053709 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87"
format_stamp: "Formatted at 2026-06-20 23:22:29 on dist-test-slave-5kzz"
server_key: "07cdaa2ccf3adfbb66d29b6c2b80029a"
server_key_iv: "6038452702e83928800e857bcccc1780"
server_key_version: "encryptionkey@0"
I20260620 23:22:29.054452 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87"
format_stamp: "Formatted at 2026-06-20 23:22:29 on dist-test-slave-5kzz"
server_key: "07cdaa2ccf3adfbb66d29b6c2b80029a"
server_key_iv: "6038452702e83928800e857bcccc1780"
server_key_version: "encryptionkey@0"
I20260620 23:22:29.059417 8640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 23:22:29.062608 8655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:29.064119 8640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:22:29.064314 8640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87"
format_stamp: "Formatted at 2026-06-20 23:22:29 on dist-test-slave-5kzz"
server_key: "07cdaa2ccf3adfbb66d29b6c2b80029a"
server_key_iv: "6038452702e83928800e857bcccc1780"
server_key_version: "encryptionkey@0"
I20260620 23:22:29.064508 8640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:29.088265 8640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:29.091956 8640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:29.092258 8640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:29.110527 8640 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45519
I20260620 23:22:29.110564 8706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45519 every 8 connection(s)
I20260620 23:22:29.111776 8640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:29.115236 8707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:29.116452 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 8640
I20260620 23:22:29.116678 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:22:29.117312 388 external_mini_cluster.cc:1428] Setting key 2de78006e510f5914cf8b14601aa28b0
I20260620 23:22:29.126648 8707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87: Bootstrap starting.
I20260620 23:22:29.129657 8707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87: Neither blocks nor log segments found. Creating new log.
Jun 20 23:22:29 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:22:29.130892 8707 log.cc:824] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:29.133790 8707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87: No bootstrap required, opened a new log
I20260620 23:22:29.138341 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:29.119222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.0.97.62:45519)
0620 23:22:29.119949 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:29.119963 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:29.120000 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:22:29.121477 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:29.121499 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:29.121848 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:29.122217 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:29.123660 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:29.124960 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:29.126206 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:29.126966 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:29.127146 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:29.131265 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:29.131271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:29.131284 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:29.131315 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:29.134565 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:29.135326 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:29.135332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:29.135337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:29.135497 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:29.135858 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:29.135861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:29.135863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:29.136403 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:29.136620 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:29.137042 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 23:22:29.137359 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":333,"threads_started":1}
I20260620 23:22:29.141917 8707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45519 } }
I20260620 23:22:29.142236 8707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:29.142350 8707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe17ba5fa7b4200bb4c0c70e9d58e87, State: Initialized, Role: FOLLOWER
I20260620 23:22:29.142941 8707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45519 } }
I20260620 23:22:29.143177 8707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:29.143298 8707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:29.143468 8707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:29.145555 8707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45519 } }
I20260620 23:22:29.146049 8707 leader_election.cc:302] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fe17ba5fa7b4200bb4c0c70e9d58e87; no voters:
I20260620 23:22:29.146554 8707 leader_election.cc:288] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:22:29.146687 8712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:29.147380 8712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 1 LEADER]: Becoming Leader. State: Replica: 9fe17ba5fa7b4200bb4c0c70e9d58e87, State: Running, Role: LEADER
I20260620 23:22:29.147876 8712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45519 } }
I20260620 23:22:29.148545 8707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:29.150933 8713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45519 } } }
I20260620 23:22:29.151330 8713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:29.151808 8714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fe17ba5fa7b4200bb4c0c70e9d58e87. Latest consensus state: current_term: 1 leader_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45519 } } }
I20260620 23:22:29.152163 8714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:29.155776 8719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:29.160897 8719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:29.168826 8719 catalog_manager.cc:1347] Generated new cluster ID: eab4ed5d383f48b785f7d760e487f706
I20260620 23:22:29.168947 8719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:29.191015 8719 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:22:29.192247 8719 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:29.201651 8719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87: Generated new TSK 0
I20260620 23:22:29.202406 8719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:22:29.262244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45519
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:29.586833 8733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:29.587075 8733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:29.587216 8733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:29.593953 8733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:29.594039 8733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:29.594223 8733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:22:29.603312 8733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:29.604399 8733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:29.605566 8733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:29.617830 8739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:29.618072 8738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:29.618043 8741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:29.618803 8733 server_base.cc:1034] running on GCE node
I20260620 23:22:29.619422 8733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:29.620291 8733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:29.621503 8733 hybrid_clock.cc:648] HybridClock initialized: now 1781997749621515 us; error 104 us; skew 500 ppm
I20260620 23:22:29.621805 8733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:29 dist-test-slave-5kzz krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:29.629526 8733 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:22:29.630834 8733 webserver.cc:460] Webserver started at http://127.0.97.1:44533/ using document root <none> and password file <none>
I20260620 23:22:29.631441 8733 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:29.631531 8733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:29.631763 8733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:29.634495 8733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "21c6063c0fed427a80526805d457c6f0"
format_stamp: "Formatted at 2026-06-20 23:22:29 on dist-test-slave-5kzz"
server_key: "2d680f115f1396757a0cec429047121f"
server_key_iv: "92b02e38a47383665f0dc6095f173161"
server_key_version: "encryptionkey@0"
I20260620 23:22:29.635267 8733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "21c6063c0fed427a80526805d457c6f0"
format_stamp: "Formatted at 2026-06-20 23:22:29 on dist-test-slave-5kzz"
server_key: "2d680f115f1396757a0cec429047121f"
server_key_iv: "92b02e38a47383665f0dc6095f173161"
server_key_version: "encryptionkey@0"
I20260620 23:22:29.640163 8733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:22:29.643241 8748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:29.644454 8733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:29.644618 8733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "21c6063c0fed427a80526805d457c6f0"
format_stamp: "Formatted at 2026-06-20 23:22:29 on dist-test-slave-5kzz"
server_key: "2d680f115f1396757a0cec429047121f"
server_key_iv: "92b02e38a47383665f0dc6095f173161"
server_key_version: "encryptionkey@0"
I20260620 23:22:29.644814 8733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:29.681028 8733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:29.685285 8733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:29.685593 8733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:29.686509 8733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:29.687809 8733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:29.687885 8733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:29.688068 8733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:29.688120 8733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:29.719573 8733 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34715
I20260620 23:22:29.719655 8860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34715 every 8 connection(s)
I20260620 23:22:29.720832 8733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:22:29.722942 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 8733
I20260620 23:22:29.723172 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:22:29.723927 388 external_mini_cluster.cc:1428] Setting key 0742253b7539bc5f5026c668ba6d3835
Jun 20 23:22:29 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:22:29.743526 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:29.723383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40277 (local address 127.0.97.62:45519)
0620 23:22:29.723897 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:29.723905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:29.724950 (+ 1045us) server_negotiation.cc:407] Connection header received
0620 23:22:29.727327 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:29.727331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:29.727389 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:29.727477 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:29.729460 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:29.730142 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:29.732616 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:29.733009 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:29.733111 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:29.736733 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:29.736742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:29.736750 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:29.736791 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:29.740325 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:29.741015 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:29.741023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:29.741030 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:29.741162 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:29.741594 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:29.741602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:29.741607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:29.741964 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:29.742113 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:29.742787 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 23:22:29.743025 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":253,"threads_started":1}
I20260620 23:22:29.744225 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:29.723975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:40277)
0620 23:22:29.724769 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:29.724815 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:29.727045 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:29.727742 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:29.727757 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:29.728531 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:29.729286 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:29.729318 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:29.730302 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:29.730310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:29.732458 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:29.732474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:29.733239 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:29.733248 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:29.733455 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:29.734147 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:29.734168 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:29.736508 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:29.740455 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:29.740462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:29.740474 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:29.740887 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:29.741262 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:29.741265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:29.741267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:29.741456 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:29.742078 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:29.742084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:29.742371 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 23:22:29.742596 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 23:22:29.743015 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":297,"threads_started":1}
I20260620 23:22:29.745853 8861 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45519
I20260620 23:22:29.746186 8861 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:29.747189 8861 heartbeater.cc:511] Master 127.0.97.62:45519 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:22:29.749714 8672 ts_manager.cc:194] Registered new tserver with Master: 21c6063c0fed427a80526805d457c6f0 (127.0.97.1:34715)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:22:29.751834 8672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:22:29.773425 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45519
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:30.078322 8867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:30.078588 8867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:30.078753 8867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:30.085829 8867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:30.085924 8867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:30.086062 8867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:22:30.095021 8867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:30.096304 8867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:30.097476 8867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:30.109404 8873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:30.109457 8875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:30.109483 8872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:30.110337 8867 server_base.cc:1034] running on GCE node
I20260620 23:22:30.115334 8867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:30.116281 8867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:30.117496 8867 hybrid_clock.cc:648] HybridClock initialized: now 1781997750117449 us; error 39 us; skew 500 ppm
I20260620 23:22:30.117798 8867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:30 dist-test-slave-5kzz krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:30.121246 8867 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:22:30.125789 8867 webserver.cc:460] Webserver started at http://127.0.97.2:43177/ using document root <none> and password file <none>
I20260620 23:22:30.126400 8867 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:30.126536 8867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:30.126863 8867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:30.129475 8867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "1dd7b6c7a67e4b1c834340fc3c947d23"
format_stamp: "Formatted at 2026-06-20 23:22:30 on dist-test-slave-5kzz"
server_key: "1c0b3c748e91123f70f070c0fbec6441"
server_key_iv: "c3cf1139bb8233f3068c173d120e06db"
server_key_version: "encryptionkey@0"
I20260620 23:22:30.130226 8867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1dd7b6c7a67e4b1c834340fc3c947d23"
format_stamp: "Formatted at 2026-06-20 23:22:30 on dist-test-slave-5kzz"
server_key: "1c0b3c748e91123f70f070c0fbec6441"
server_key_iv: "c3cf1139bb8233f3068c173d120e06db"
server_key_version: "encryptionkey@0"
I20260620 23:22:30.135052 8867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260620 23:22:30.138134 8882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:30.139396 8867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:30.139569 8867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "1dd7b6c7a67e4b1c834340fc3c947d23"
format_stamp: "Formatted at 2026-06-20 23:22:30 on dist-test-slave-5kzz"
server_key: "1c0b3c748e91123f70f070c0fbec6441"
server_key_iv: "c3cf1139bb8233f3068c173d120e06db"
server_key_version: "encryptionkey@0"
I20260620 23:22:30.139767 8867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:30.159483 8867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:30.163440 8867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:30.163707 8867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:30.164577 8867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:30.165747 8867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:30.165818 8867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:30.165984 8867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:30.166036 8867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:30.197171 8867 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45229
I20260620 23:22:30.197223 8994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45229 every 8 connection(s)
I20260620 23:22:30.198499 8867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:22:30.202309 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 8867
I20260620 23:22:30.202468 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:22:30.202924 388 external_mini_cluster.cc:1428] Setting key 3621165ea4bb38155ada5aead1c64e6b
Jun 20 23:22:30 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:22:30.217988 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:30.200942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50389 (local address 127.0.97.62:45519)
0620 23:22:30.201097 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:30.201102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:30.202248 (+ 1146us) server_negotiation.cc:407] Connection header received
0620 23:22:30.204413 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:30.204417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:30.204511 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:30.204611 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:30.205988 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:30.206655 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:30.208115 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:30.208552 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:30.208677 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:30.212543 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:30.212548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:30.212552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:30.212580 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:30.215057 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:30.215890 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:30.215894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:30.215896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:30.216004 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:30.216463 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:30.216466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:30.216468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:30.216694 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:30.216793 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:30.217617 (+ 824us) server_negotiation.cc:299] Negotiation successful
0620 23:22:30.217772 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:22:30.219641 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:30.201418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:50389)
0620 23:22:30.202100 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:30.202140 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:30.204150 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:30.204854 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:30.204870 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:30.205326 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:30.205836 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:30.205851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:30.206823 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:30.206828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:30.207982 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:30.207992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:30.208770 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:30.208776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:30.208988 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:30.209645 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:30.209670 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:30.212342 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:30.215180 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:30.215191 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:30.215209 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:30.215784 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:30.216115 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:30.216122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:30.216128 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:30.216362 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:30.216812 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:30.216822 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:30.217260 (+ 438us) client_negotiation.cc:769] Sending connection context
0620 23:22:30.217557 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 23:22:30.218091 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":296,"threads_started":1}
I20260620 23:22:30.221117 8995 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45519
I20260620 23:22:30.221400 8995 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:30.222060 8995 heartbeater.cc:511] Master 127.0.97.62:45519 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:22:30.223562 8672 ts_manager.cc:194] Registered new tserver with Master: 1dd7b6c7a67e4b1c834340fc3c947d23 (127.0.97.2:45229)
I20260620 23:22:30.224403 8672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:22:30.248279 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45519
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:30.552706 9000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:30.553002 9000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:30.553166 9000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:30.560331 9000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:30.560475 9000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:30.560648 9000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:22:30.570032 9000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:30.571377 9000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:30.572669 9000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:30.585040 9005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:30.585091 9006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:30.585060 9008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:30.586211 9000 server_base.cc:1034] running on GCE node
I20260620 23:22:30.586931 9000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:30.587729 9000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:30.588948 9000 hybrid_clock.cc:648] HybridClock initialized: now 1781997750588910 us; error 51 us; skew 500 ppm
I20260620 23:22:30.589241 9000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:30 dist-test-slave-5kzz krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:30.592161 9000 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:22:30.593351 9000 webserver.cc:460] Webserver started at http://127.0.97.3:34395/ using document root <none> and password file <none>
I20260620 23:22:30.593942 9000 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:30.594034 9000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:30.594269 9000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:30.597164 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "bc8f68c1ff3d40d3b04604d57da2924f"
format_stamp: "Formatted at 2026-06-20 23:22:30 on dist-test-slave-5kzz"
server_key: "7c84bad7757edad18db03bd5d6b9b19d"
server_key_iv: "eb40eb22b3c2a2a48cc00fcaf7518c57"
server_key_version: "encryptionkey@0"
I20260620 23:22:30.597854 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bc8f68c1ff3d40d3b04604d57da2924f"
format_stamp: "Formatted at 2026-06-20 23:22:30 on dist-test-slave-5kzz"
server_key: "7c84bad7757edad18db03bd5d6b9b19d"
server_key_iv: "eb40eb22b3c2a2a48cc00fcaf7518c57"
server_key_version: "encryptionkey@0"
I20260620 23:22:30.602320 9000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260620 23:22:30.605428 9015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:30.606586 9000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 23:22:30.606777 9000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "bc8f68c1ff3d40d3b04604d57da2924f"
format_stamp: "Formatted at 2026-06-20 23:22:30 on dist-test-slave-5kzz"
server_key: "7c84bad7757edad18db03bd5d6b9b19d"
server_key_iv: "eb40eb22b3c2a2a48cc00fcaf7518c57"
server_key_version: "encryptionkey@0"
I20260620 23:22:30.606936 9000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:30.628273 9000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:30.631803 9000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:30.632014 9000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:30.632794 9000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:30.633975 9000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:30.634048 9000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:30.634125 9000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:30.634168 9000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:30.665823 9000 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39709
I20260620 23:22:30.665872 9127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39709 every 8 connection(s)
I20260620 23:22:30.667157 9000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:22:30.677389 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 9000
I20260620 23:22:30.677579 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:22:30.677989 388 external_mini_cluster.cc:1428] Setting key 56ae90fd5f54f0fba79a11fffc939bb7
Jun 20 23:22:30 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:22:30.685065 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:30.669694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37615 (local address 127.0.97.62:45519)
0620 23:22:30.669855 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:30.669859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:30.670956 (+ 1097us) server_negotiation.cc:407] Connection header received
0620 23:22:30.672578 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:30.672582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:30.672642 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:30.672732 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:30.674101 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:30.674897 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:30.676298 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:30.676643 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:30.676729 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:30.680149 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:30.680167 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:30.680170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:30.680204 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:30.682782 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:30.683349 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:30.683352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:30.683354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:30.683458 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:30.683796 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:30.683799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:30.683801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:30.684044 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:30.684146 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:30.684699 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 23:22:30.684834 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:22:30.685913 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:30.670118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:37615)
0620 23:22:30.670821 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:30.670856 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:30.672358 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:30.672914 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:30.672920 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:30.673423 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:30.673957 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:30.673972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:30.675015 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:30.675025 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:30.676181 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:30.676191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:30.676804 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:30.676809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:30.676987 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:30.677641 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:30.677686 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:30.679966 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:30.682895 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:30.682901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:30.682911 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:30.683262 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:30.683547 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:30.683549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:30.683552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:30.683706 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:30.684140 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:30.684145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:30.684397 (+ 252us) client_negotiation.cc:769] Sending connection context
0620 23:22:30.684615 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 23:22:30.684926 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":263,"threads_started":1}
I20260620 23:22:30.687193 9128 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45519
I20260620 23:22:30.687419 9128 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:30.687976 9128 heartbeater.cc:511] Master 127.0.97.62:45519 requested a full tablet report, sending...
I20260620 23:22:30.689209 8672 ts_manager.cc:194] Registered new tserver with Master: bc8f68c1ff3d40d3b04604d57da2924f (127.0.97.3:39709)
I20260620 23:22:30.689966 8672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37615
I20260620 23:22:30.692698 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260620 23:22:30.755457 8861 heartbeater.cc:503] Master 127.0.97.62:45519 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 23:22:30.807426 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 8733
Jun 20 23:22:30 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:22:30.823555 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:30.808490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.0.97.1:34715)
0620 23:22:30.809080 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:30.809087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:30.809116 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:22:30.809214 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:30.809222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:30.809441 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:30.809633 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:30.810850 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:30.811845 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:30.813002 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:30.813506 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:30.813645 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:30.817500 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:30.817512 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:30.817526 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:30.817557 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:30.820558 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:30.821374 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:30.821383 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:30.821385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:30.821508 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:30.821904 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:30.821912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:30.821915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:30.822248 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:30.822595 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:30.822928 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 23:22:30.823189 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":288,"threads_started":1}
I20260620 23:22:30.824834 8856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:30.904146 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 8733
I20260620 23:22:30.919927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 8867
Jun 20 23:22:30 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:22:30.934027 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:30.920801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.0.97.2:45229)
0620 23:22:30.921333 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:30.921340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:30.921364 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:22:30.921418 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:30.921424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:30.921615 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:30.921751 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:30.922852 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:30.923775 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:30.924873 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:30.925368 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:30.925458 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:30.929340 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:30.929349 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:30.929362 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:30.929397 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:30.931838 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:30.932381 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:30.932388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:30.932390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:30.932501 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:30.932785 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:30.932788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:30.932790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:30.933115 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:30.933323 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:30.933554 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 23:22:30.933748 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":299,"threads_started":1}
I20260620 23:22:30.935285 8990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:31.014207 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 8867
I20260620 23:22:31.029464 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 9000
Jun 20 23:22:31 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:22:31.045895 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:31.030474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.0.97.3:39709)
0620 23:22:31.031123 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:31.031129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:31.031157 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:22:31.031211 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:31.031217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:31.031420 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:31.031586 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:31.032855 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:31.033855 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:31.035158 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:31.035685 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:31.035816 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:31.039899 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:31.039908 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:31.039921 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:31.039953 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:31.042606 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:31.043464 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:31.043509 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:31.043515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:31.043686 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:31.044184 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:31.044189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:31.044192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:31.044673 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:31.044983 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:31.045224 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 23:22:31.045484 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":331,"threads_started":1}
I20260620 23:22:31.047178 9123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:31.130743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 9000
I20260620 23:22:31.146042 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 8640
I20260620 23:22:31.147217 8702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:31.244282 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 8640
I20260620 23:22:31.263880 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45519
--webserver_interface=127.0.97.62
--webserver_port=35121
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:31.568648 9145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:31.568917 9145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:31.569072 9145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:31.579559 9145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:31.579727 9145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:31.579808 9145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:31.579892 9145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:31.579958 9145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:22:31.591073 9145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:31.592379 9145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:31.593698 9145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:31.605100 9150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:31.605137 9153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:31.605159 9151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:31.605976 9145 server_base.cc:1034] running on GCE node
I20260620 23:22:31.606724 9145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:31.608008 9145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:31.609227 9145 hybrid_clock.cc:648] HybridClock initialized: now 1781997751609180 us; error 59 us; skew 500 ppm
I20260620 23:22:31.609553 9145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:31 dist-test-slave-5kzz krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:31.612417 9145 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:22:31.613778 9145 webserver.cc:460] Webserver started at http://127.0.97.62:35121/ using document root <none> and password file <none>
I20260620 23:22:31.614418 9145 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:31.614573 9145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:31.619628 9145 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:22:31.622223 9160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:31.623468 9145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:22:31.623642 9145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87"
format_stamp: "Formatted at 2026-06-20 23:22:29 on dist-test-slave-5kzz"
server_key: "07cdaa2ccf3adfbb66d29b6c2b80029a"
server_key_iv: "6038452702e83928800e857bcccc1780"
server_key_version: "encryptionkey@0"
I20260620 23:22:31.624545 9145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:31.635875 9145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:31.639340 9145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:31.639585 9145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:31.656980 9145 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45519
I20260620 23:22:31.657011 9211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45519 every 8 connection(s)
I20260620 23:22:31.658349 9145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:31.663916 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 9145
I20260620 23:22:31.664824 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34715
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45519
--webserver_port=44533
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:22:31.665285 9212 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:22:31.670254 9212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87: Bootstrap starting.
I20260620 23:22:31.689229 9212 log.cc:824] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:31.697543 9212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:22:31.698041 9212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87: Bootstrap complete.
I20260620 23:22:31.706506 9212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45519 } }
I20260620 23:22:31.707381 9212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe17ba5fa7b4200bb4c0c70e9d58e87, State: Initialized, Role: FOLLOWER
I20260620 23:22:31.707901 9212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45519 } }
I20260620 23:22:31.708130 9212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:31.708242 9212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:31.708400 9212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:22:31.711989 9212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45519 } }
I20260620 23:22:31.712520 9212 leader_election.cc:302] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fe17ba5fa7b4200bb4c0c70e9d58e87; no voters:
I20260620 23:22:31.713131 9212 leader_election.cc:288] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:22:31.713260 9216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:22:31.714627 9216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [term 2 LEADER]: Becoming Leader. State: Replica: 9fe17ba5fa7b4200bb4c0c70e9d58e87, State: Running, Role: LEADER
I20260620 23:22:31.715193 9216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45519 } }
I20260620 23:22:31.715380 9212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:31.717422 9218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fe17ba5fa7b4200bb4c0c70e9d58e87. Latest consensus state: current_term: 2 leader_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45519 } } }
I20260620 23:22:31.717767 9218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:31.718288 9217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe17ba5fa7b4200bb4c0c70e9d58e87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45519 } } }
I20260620 23:22:31.718582 9217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:31.727305 9223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:31.732807 9223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:31.734212 9223 catalog_manager.cc:1259] Loaded cluster ID: eab4ed5d383f48b785f7d760e487f706
I20260620 23:22:31.734371 9223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:31.738255 9223 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:31.739971 9223 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9fe17ba5fa7b4200bb4c0c70e9d58e87: Loaded TSK: 0
I20260620 23:22:31.741549 9223 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:22:31.987816 9214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:31.988102 9214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:31.988251 9214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:31.995173 9214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:31.995303 9214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:31.995464 9214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:22:32.004614 9214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44533
--tserver_master_addrs=127.0.97.62:45519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:32.005878 9214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:32.007164 9214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:32.021227 9239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:32.021204 9238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:32.026244 9241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:32.026823 9214 server_base.cc:1034] running on GCE node
I20260620 23:22:32.027416 9214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:32.028362 9214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:32.029595 9214 hybrid_clock.cc:648] HybridClock initialized: now 1781997752029540 us; error 49 us; skew 500 ppm
I20260620 23:22:32.029927 9214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997752, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:32.033120 9214 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 23:22:32.034729 9214 webserver.cc:460] Webserver started at http://127.0.97.1:44533/ using document root <none> and password file <none>
I20260620 23:22:32.035454 9214 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:32.035614 9214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:32.040769 9214 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 23:22:32.043460 9248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:32.044771 9214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:22:32.044935 9214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "21c6063c0fed427a80526805d457c6f0"
format_stamp: "Formatted at 2026-06-20 23:22:29 on dist-test-slave-5kzz"
server_key: "2d680f115f1396757a0cec429047121f"
server_key_iv: "92b02e38a47383665f0dc6095f173161"
server_key_version: "encryptionkey@0"
I20260620 23:22:32.045861 9214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:32.068804 9214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:32.072623 9214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:32.072911 9214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:32.073817 9214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:32.075145 9214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:32.075266 9214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:32.075387 9214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:32.075470 9214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:32.106652 9214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34715
I20260620 23:22:32.106771 9360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34715 every 8 connection(s)
I20260620 23:22:32.108081 9214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:22:32.117292 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 9214
I20260620 23:22:32.117935 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45229
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45519
--webserver_port=43177
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:32.126313 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:32.110919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:56025)
0620 23:22:32.111747 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:32.111790 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:32.113403 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:32.114654 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:32.114661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:32.115182 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:32.115756 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.115774 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.117688 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.117692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.119024 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.119036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.120423 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.120427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.120615 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.121244 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:32.121264 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:32.125113 (+ 3849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":355,"threads_started":1}
W20260620 23:22:32.126358 9363 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:32.110854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56025 (local address 127.0.97.62:45519)
0620 23:22:32.111620 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:32.111630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:32.111901 (+ 271us) server_negotiation.cc:407] Connection header received
0620 23:22:32.113914 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:32.113927 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:32.114248 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:32.114526 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:32.115927 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.117540 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.119153 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.120235 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.120415 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.125447 (+ 5032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56025: BlockingRecv error: recv got EOF from 127.0.97.1:56025 (error 108)
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":343,"threads_started":1}
W20260620 23:22:32.127238 9361 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45519 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45519: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:32.135382 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:32.127855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:46743)
0620 23:22:32.128108 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:32.128128 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:32.128216 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:32.128503 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:32.128507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:32.128848 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:32.129010 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.129033 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.130060 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.130065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.131090 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.131101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.131714 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.131719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.131878 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.132442 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:32.132456 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:32.135119 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115}
W20260620 23:22:32.135568 9363 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:32.128063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46743 (local address 127.0.97.62:45519)
0620 23:22:32.128213 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:32.128218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:32.128234 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:22:32.128316 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:32.128321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:32.128399 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:32.128484 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:32.129176 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.129926 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.131214 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.131585 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.131705 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.135277 (+ 3572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46743: BlockingRecv error: recv got EOF from 127.0.97.1:46743 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:32.143381 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:32.136317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:49883)
0620 23:22:32.136483 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:32.136506 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:32.136615 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:32.136916 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:32.136921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:32.137204 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:32.137321 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.137328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.138254 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.138259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.139191 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.139201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.139861 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.139866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.140022 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.140536 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:32.140552 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:32.143072 (+ 2520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:22:32.143653 9361 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:22:32.143711 9363 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:32.136498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49883 (local address 127.0.97.62:45519)
0620 23:22:32.136657 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:32.136664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:32.136681 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:22:32.136723 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:32.136728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:32.136799 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:32.136897 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:32.137446 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.138123 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.139323 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.139729 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.139854 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.143227 (+ 3373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49883: BlockingRecv error: recv got EOF from 127.0.97.1:49883 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 23:22:32.423448 9365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:32.423731 9365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:32.423887 9365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:32.430968 9365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:32.431097 9365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:32.431250 9365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:22:32.440532 9365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43177
--tserver_master_addrs=127.0.97.62:45519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:32.441704 9365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:32.442945 9365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:32.455403 9370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:32.455459 9373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:32.455480 9371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:32.456368 9365 server_base.cc:1034] running on GCE node
I20260620 23:22:32.456840 9365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:32.457612 9365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:32.458782 9365 hybrid_clock.cc:648] HybridClock initialized: now 1781997752458731 us; error 53 us; skew 500 ppm
I20260620 23:22:32.459079 9365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997752, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:32.464571 9365 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 23:22:32.465895 9365 webserver.cc:460] Webserver started at http://127.0.97.2:43177/ using document root <none> and password file <none>
I20260620 23:22:32.466512 9365 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:32.466661 9365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:32.471630 9365 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:22:32.474023 9380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:32.475195 9365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 23:22:32.475348 9365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "1dd7b6c7a67e4b1c834340fc3c947d23"
format_stamp: "Formatted at 2026-06-20 23:22:30 on dist-test-slave-5kzz"
server_key: "1c0b3c748e91123f70f070c0fbec6441"
server_key_iv: "c3cf1139bb8233f3068c173d120e06db"
server_key_version: "encryptionkey@0"
I20260620 23:22:32.476235 9365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:32.496630 9365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:32.501995 9365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:32.502251 9365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:32.503234 9365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:32.505030 9365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:32.505180 9365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:32.505300 9365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:32.505396 9365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:32.541460 9365 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45229
I20260620 23:22:32.541548 9492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45229 every 8 connection(s)
I20260620 23:22:32.542948 9365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:22:32.550056 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 9365
I20260620 23:22:32.550814 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39709
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45519
--webserver_port=34395
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:22:32.558629 9363 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:32.545347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38059 (local address 127.0.97.62:45519)
0620 23:22:32.545537 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:32.545541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:32.546608 (+ 1067us) server_negotiation.cc:407] Connection header received
0620 23:22:32.548274 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:32.548278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:32.548338 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:32.548420 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:32.549777 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.550849 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.552582 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.553265 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.553394 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.558454 (+ 5060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38059: BlockingRecv error: recv got EOF from 127.0.97.2:38059 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:22:32.559548 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:32.545814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:38059)
0620 23:22:32.546448 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:32.546488 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:32.548049 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:32.548621 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:32.548628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:32.549099 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:32.549622 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.549638 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.551117 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.551125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.552439 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.552450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.553449 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.553453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.553654 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.554288 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:32.554309 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:32.558322 (+ 4013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":261,"threads_started":1}
W20260620 23:22:32.560135 9493 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45519 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45519: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:32.567718 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:32.560660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:56419)
0620 23:22:32.560797 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:32.560815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:32.560897 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:32.561153 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:32.561157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:32.561464 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:32.561597 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.561603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.562578 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.562586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.563455 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.563464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.564093 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.564098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.564246 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.564790 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:32.564812 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:32.567469 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:22:32.568089 9363 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:32.560748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56419 (local address 127.0.97.62:45519)
0620 23:22:32.560901 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:32.560906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:32.560922 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:22:32.560972 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:32.560976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:32.561045 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:32.561137 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:32.561722 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.562406 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.563588 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.563952 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.564086 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.567651 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56419: BlockingRecv error: recv got EOF from 127.0.97.2:56419 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:32.575940 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:32.568263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:44543)
0620 23:22:32.568401 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:32.568418 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:32.568501 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:32.569017 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:32.569020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:32.569260 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:32.569384 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.569390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.570884 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.570888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.571723 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.571732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.572631 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.572634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.572740 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.573259 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:32.573274 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:32.575764 (+ 2490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 23:22:32.576205 9363 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:32.568508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44543 (local address 127.0.97.62:45519)
0620 23:22:32.568717 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:32.568724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:32.568744 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:22:32.568799 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:32.568805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:32.568882 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:32.568995 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:32.569510 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.570747 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.571855 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.572510 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.572623 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.575973 (+ 3350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44543: BlockingRecv error: recv got EOF from 127.0.97.2:44543 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 23:22:32.576359 9493 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:22:32.858481 9496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:32.858850 9496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:32.859012 9496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:32.865604 9496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:32.865729 9496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:32.865885 9496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:22:32.874902 9496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34395
--tserver_master_addrs=127.0.97.62:45519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:32.876122 9496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:32.877350 9496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:32.889984 9501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:32.889997 9504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:32.890036 9502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:32.890722 9496 server_base.cc:1034] running on GCE node
I20260620 23:22:32.891291 9496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:32.892104 9496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:32.893314 9496 hybrid_clock.cc:648] HybridClock initialized: now 1781997752893281 us; error 48 us; skew 500 ppm
I20260620 23:22:32.893674 9496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997752, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:32.896535 9496 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 23:22:32.897799 9496 webserver.cc:460] Webserver started at http://127.0.97.3:34395/ using document root <none> and password file <none>
I20260620 23:22:32.898447 9496 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:32.898600 9496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:32.903455 9496 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:22:32.905862 9511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:32.906975 9496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:32.907141 9496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "bc8f68c1ff3d40d3b04604d57da2924f"
format_stamp: "Formatted at 2026-06-20 23:22:30 on dist-test-slave-5kzz"
server_key: "7c84bad7757edad18db03bd5d6b9b19d"
server_key_iv: "eb40eb22b3c2a2a48cc00fcaf7518c57"
server_key_version: "encryptionkey@0"
I20260620 23:22:32.907981 9496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:32.930465 9496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:32.933934 9496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:32.934247 9496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:32.935151 9496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:32.936348 9496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:32.936470 9496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:32.936601 9496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:32.936688 9496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:32.967566 9496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39709
I20260620 23:22:32.967650 9623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39709 every 8 connection(s)
I20260620 23:22:32.969043 9496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:22:32.971052 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 9496
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:22:32.985642 9363 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:32.971627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52651 (local address 127.0.97.62:45519)
0620 23:22:32.971838 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:32.971844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:32.973458 (+ 1614us) server_negotiation.cc:407] Connection header received
0620 23:22:32.975618 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:32.975621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:32.975674 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:32.975757 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:32.977090 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.978013 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.979683 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.980403 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.980549 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.985404 (+ 4855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52651: BlockingRecv error: recv got EOF from 127.0.97.3:52651 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:22:32.986316 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:32.972198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:52651)
0620 23:22:32.973313 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:32.973349 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:32.975342 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:32.975992 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:32.975999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:32.976416 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:32.976935 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.976956 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.978157 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.978164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.979540 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.979559 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.980636 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.980643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.980856 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.981531 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:32.981550 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:32.985232 (+ 3682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":586,"threads_started":1}
I20260620 23:22:32.986356 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:32.971980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.0.97.62:45519)
0620 23:22:32.972714 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:32.972718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:32.972734 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:22:32.972879 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:32.972886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:32.972976 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:32.973105 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:32.974660 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.975812 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.976897 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.977374 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.977458 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.980455 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:32.980463 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:32.980474 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:32.980505 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:32.983633 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:32.984159 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:32.984164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:32.984169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:32.984300 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:32.984658 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:32.984665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:32.984669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:32.985056 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:32.985359 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:32.985755 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 23:22:32.986088 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":549,"threads_started":1}
W20260620 23:22:32.987418 9624 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45519 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45519: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:32.994889 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:32.987916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:44489)
0620 23:22:32.988046 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:32.988062 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:32.988143 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:32.988446 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:32.988450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:32.989062 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:32.989183 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.989190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.990204 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.990209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.991080 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.991090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.991709 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.991714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.991862 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.992348 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:32.992363 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:32.994652 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 23:22:32.995106 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:32.987982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44489 (local address 127.0.97.62:45519)
0620 23:22:32.988124 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:32.988128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:32.988141 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:22:32.988265 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:32.988270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:32.988342 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:32.988428 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:32.989308 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.990076 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.991212 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.991570 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.991688 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.994867 (+ 3179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44489: BlockingRecv error: recv got EOF from 127.0.97.3:44489 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:32 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:33.002043 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:32.995426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:48611)
0620 23:22:32.995615 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:32.995633 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:32.995720 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:32.995980 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:32.995984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:32.996250 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:32.996368 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.996375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.997245 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.997250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.998081 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:32.998091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.998766 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.998771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:32.998945 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:32.999369 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:32.999384 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:33.001755 (+ 2371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:22:33.002180 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:32.995515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48611 (local address 127.0.97.62:45519)
0620 23:22:32.995737 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:32.995743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:32.995770 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:22:32.995812 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:32.995817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:32.995871 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:32.995962 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:32.996491 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.997107 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.998214 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:32.998602 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:32.998776 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:33.001910 (+ 3134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48611: BlockingRecv error: recv got EOF from 127.0.97.3:48611 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260620 23:22:33.002313 9624 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 23:22:33 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:33 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:33.152540 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:33.144225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:34601)
0620 23:22:33.144689 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:33.144703 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:33.144848 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:33.145178 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:33.145183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:33.145625 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:33.145882 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:33.145891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:33.146930 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:33.146936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:33.147829 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:33.147838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:33.148560 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:33.148565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:33.148733 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:33.149252 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:33.149269 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:33.152300 (+ 3031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":244,"threads_started":1}
W20260620 23:22:33.152724 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:33.144292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34601 (local address 127.0.97.62:45519)
0620 23:22:33.144511 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:33.144519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:33.144787 (+ 268us) server_negotiation.cc:407] Connection header received
0620 23:22:33.144983 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:33.144988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:33.145057 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:33.145151 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:33.146015 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:33.146808 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:33.147979 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:33.148380 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:33.148516 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:33.152505 (+ 3989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34601: BlockingRecv error: recv got EOF from 127.0.97.1:34601 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 23:22:33 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:33 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:33.585350 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:33.576957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:34133)
0620 23:22:33.577535 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:33.577554 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:33.577693 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:33.578198 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:33.578201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:33.578596 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:33.578878 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:33.578889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:33.579923 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:33.579926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:33.580915 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:33.580927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:33.581533 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:33.581537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:33.581655 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:33.582351 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:33.582368 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:33.585071 (+ 2703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":347,"threads_started":1}
W20260620 23:22:33.585703 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:33.577061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34133 (local address 127.0.97.62:45519)
0620 23:22:33.577214 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:33.577220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:33.577621 (+ 401us) server_negotiation.cc:407] Connection header received
0620 23:22:33.577829 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:33.577834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:33.577942 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:33.578044 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:33.579004 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:33.579745 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:33.581026 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:33.581399 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:33.581485 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:33.585299 (+ 3814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34133: BlockingRecv error: recv got EOF from 127.0.97.2:34133 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 23:22:34 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:34 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:34.011304 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:34.003028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:54901)
0620 23:22:34.003472 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:34.003486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:34.003633 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:34.003959 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:34.003964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:34.004530 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:34.004722 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:34.004731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:34.005830 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:34.005837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:34.006859 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:34.006870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:34.007643 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:34.007649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:34.007835 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:34.008343 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:34.008358 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:34.010956 (+ 2598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":228,"threads_started":1}
W20260620 23:22:34.011322 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:34.003086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54901 (local address 127.0.97.62:45519)
0620 23:22:34.003286 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:34.003294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:34.003573 (+ 279us) server_negotiation.cc:407] Connection header received
0620 23:22:34.003767 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:34.003772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:34.003838 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:34.003935 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:34.004861 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:34.005700 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:34.007013 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:34.007468 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:34.007614 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:34.011125 (+ 3511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54901: BlockingRecv error: recv got EOF from 127.0.97.3:54901 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 23:22:34 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:34 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:34.161303 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:34.153665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:49315)
0620 23:22:34.154086 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:34.154101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:34.154224 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:34.154490 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:34.154493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:34.154875 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:34.155008 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:34.155016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:34.155941 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:34.155944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:34.156983 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:34.156997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:34.157814 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:34.157823 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:34.157986 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:34.158535 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:34.158552 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:34.161086 (+ 2534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":236,"threads_started":1}
W20260620 23:22:34.161629 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:34.153691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49315 (local address 127.0.97.62:45519)
0620 23:22:34.153907 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:34.153916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:34.154203 (+ 287us) server_negotiation.cc:407] Connection header received
0620 23:22:34.154315 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:34.154319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:34.154375 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:34.154452 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:34.155115 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:34.155831 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:34.157114 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:34.157648 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:34.157778 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:34.161314 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49315: BlockingRecv error: recv got EOF from 127.0.97.1:49315 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 23:22:34 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:34 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:34.594128 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:34.586414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:57405)
0620 23:22:34.586919 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:34.586934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:34.587092 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:34.587425 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:34.587430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:34.587814 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:34.587951 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:34.587958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:34.588980 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:34.588986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:34.589867 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:34.589877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:34.590547 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:34.590553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:34.590737 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:34.591310 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:34.591325 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:34.593786 (+ 2461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":285,"threads_started":1}
W20260620 23:22:34.594173 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:34.586477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57405 (local address 127.0.97.62:45519)
0620 23:22:34.586750 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:34.586758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:34.587022 (+ 264us) server_negotiation.cc:407] Connection header received
0620 23:22:34.587211 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:34.587216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:34.587291 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:34.587396 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:34.588094 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:34.588816 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:34.590008 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:34.590408 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:34.590536 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:34.593968 (+ 3432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57405: BlockingRecv error: recv got EOF from 127.0.97.2:57405 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 23:22:35 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:35 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:35.022501 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:35.012114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:53699)
0620 23:22:35.012625 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:35.012639 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:35.012739 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:35.012999 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:35.013002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:35.013369 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:35.013498 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:35.013506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:35.014430 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:35.014434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:35.015527 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:35.015539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:35.016178 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:35.016182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:35.016284 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:35.019709 (+ 3425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:35.019726 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:35.022276 (+ 2550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":283,"threads_started":1}
W20260620 23:22:35.022643 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:35.012295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53699 (local address 127.0.97.62:45519)
0620 23:22:35.012463 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:35.012467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:35.012705 (+ 238us) server_negotiation.cc:407] Connection header received
0620 23:22:35.012827 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:35.012830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:35.012884 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:35.012958 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:35.013608 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:35.014282 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:35.015645 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:35.016061 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:35.016148 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:35.022442 (+ 6294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53699: BlockingRecv error: recv got EOF from 127.0.97.3:53699 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:22:35 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:35 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:35.169483 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:35.162143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:54711)
0620 23:22:35.162562 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:35.162578 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:35.162702 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:35.162944 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:35.162947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:35.163267 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:35.163423 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:35.163431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:35.164329 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:35.164332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:35.165164 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:35.165173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:35.165739 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:35.165743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:35.165853 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:35.166359 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:35.166375 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:35.169120 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":241,"threads_started":1}
W20260620 23:22:35.169524 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:35.162238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54711 (local address 127.0.97.62:45519)
0620 23:22:35.162487 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:35.162493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:35.162654 (+ 161us) server_negotiation.cc:407] Connection header received
0620 23:22:35.162797 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:35.162800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:35.162858 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:35.162933 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:35.163548 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:35.164225 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:35.165272 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:35.165632 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:35.165715 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:35.169291 (+ 3576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54711: BlockingRecv error: recv got EOF from 127.0.97.1:54711 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jun 20 23:22:35 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:35 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:35.603350 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:35.595187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:42213)
0620 23:22:35.595660 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:35.595674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:35.595814 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:35.596177 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:35.596182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:35.596567 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:35.596719 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:35.596726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:35.597704 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:35.597712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:35.598579 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:35.598589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:35.599370 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:35.599375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:35.599565 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:35.600100 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:35.600116 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:35.603114 (+ 2998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":252,"threads_started":1}
W20260620 23:22:35.603538 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:35.595268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42213 (local address 127.0.97.62:45519)
0620 23:22:35.595492 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:35.595501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:35.595757 (+ 256us) server_negotiation.cc:407] Connection header received
0620 23:22:35.595937 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:35.595942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:35.596036 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:35.596146 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:35.596890 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:35.597569 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:35.598751 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:35.599223 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:35.599360 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:35.603294 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42213: BlockingRecv error: recv got EOF from 127.0.97.2:42213 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 23:22:36 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:36 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:36.031466 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:36.023315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:48017)
0620 23:22:36.023813 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:36.023827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:36.023972 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:36.024322 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:36.024327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:36.024676 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:36.024828 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:36.024835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:36.025842 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:36.025847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:36.026796 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:36.026807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:36.027465 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:36.027470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:36.027686 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:36.028243 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:36.028259 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:36.031200 (+ 2941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":291,"threads_started":1}
W20260620 23:22:36.031848 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:36.023408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48017 (local address 127.0.97.62:45519)
0620 23:22:36.023622 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:36.023630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:36.023903 (+ 273us) server_negotiation.cc:407] Connection header received
0620 23:22:36.024102 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:36.024107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:36.024171 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:36.024291 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:36.024966 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:36.025705 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:36.026927 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:36.027325 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:36.027456 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:36.031663 (+ 4207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48017: BlockingRecv error: recv got EOF from 127.0.97.3:48017 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 23:22:36 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:36 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:36.178486 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:36.170495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:33617)
0620 23:22:36.171043 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:36.171062 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:36.171196 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:36.171513 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:36.171517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:36.171872 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:36.172086 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:36.172097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:36.173028 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:36.173035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:36.173967 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:36.173977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:36.174751 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:36.174756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:36.174932 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:36.175534 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:36.175551 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:36.178179 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":258,"threads_started":1}
W20260620 23:22:36.178579 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:36.170587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33617 (local address 127.0.97.62:45519)
0620 23:22:36.170801 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:36.170808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:36.171162 (+ 354us) server_negotiation.cc:407] Connection header received
0620 23:22:36.171317 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:36.171323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:36.171388 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:36.171486 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:36.172220 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:36.172911 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:36.174110 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:36.174541 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:36.174726 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:36.178350 (+ 3624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33617: BlockingRecv error: recv got EOF from 127.0.97.1:33617 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:22:36 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:36 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:36.612727 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:36.604243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:56841)
0620 23:22:36.604767 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:36.604782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:36.604938 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:36.605282 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:36.605287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:36.605652 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:36.605842 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:36.605855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:36.607233 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:36.607239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:36.608188 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:36.608199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:36.608831 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:36.608836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:36.609033 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:36.609663 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:36.609680 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:36.612412 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":288,"threads_started":1}
W20260620 23:22:36.612823 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:36.604440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56841 (local address 127.0.97.62:45519)
0620 23:22:36.604698 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:36.604704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:36.604868 (+ 164us) server_negotiation.cc:407] Connection header received
0620 23:22:36.605055 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:36.605062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:36.605149 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:36.605251 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:36.606006 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:36.607097 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:36.608311 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:36.608692 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:36.608780 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:36.612594 (+ 3814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56841: BlockingRecv error: recv got EOF from 127.0.97.2:56841 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 20 23:22:37 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:37 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:37.039669 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:37.032220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:55199)
0620 23:22:37.032640 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:37.032656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:37.032758 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:37.033021 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:37.033024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:37.033344 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:37.033483 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:37.033490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:37.034394 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:37.034397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:37.035327 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:37.035338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:37.035924 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:37.035927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:37.036054 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:37.036579 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:37.036595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:37.039332 (+ 2737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":223,"threads_started":1}
W20260620 23:22:37.039846 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:37.032294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55199 (local address 127.0.97.62:45519)
0620 23:22:37.032538 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:37.032544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:37.032728 (+ 184us) server_negotiation.cc:407] Connection header received
0620 23:22:37.032848 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:37.032851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:37.032908 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:37.032982 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:37.033599 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:37.034295 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:37.035442 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:37.035813 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:37.035898 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:37.039524 (+ 3626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55199: BlockingRecv error: recv got EOF from 127.0.97.3:55199 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 23:22:37 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:37 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:37.187594 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:37.179641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:34233)
0620 23:22:37.180231 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:37.180258 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:37.180421 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:37.180744 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:37.180749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:37.181165 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:37.181308 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:37.181316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:37.182281 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:37.182288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:37.183230 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:37.183240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:37.183917 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:37.183922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:37.184071 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:37.184669 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:37.184684 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:37.187263 (+ 2579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":312,"threads_started":1}
W20260620 23:22:37.187615 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:37.179711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34233 (local address 127.0.97.62:45519)
0620 23:22:37.179938 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:37.179943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:37.180351 (+ 408us) server_negotiation.cc:407] Connection header received
0620 23:22:37.180538 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:37.180543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:37.180607 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:37.180699 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:37.181458 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:37.182156 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:37.183361 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:37.183791 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:37.183908 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:37.187424 (+ 3516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34233: BlockingRecv error: recv got EOF from 127.0.97.1:34233 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 20 23:22:37 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:37 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:37.621870 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:37.613742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:43419)
0620 23:22:37.614274 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:37.614289 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:37.614418 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:37.614826 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:37.614830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:37.615401 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:37.615566 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:37.615574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:37.616592 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:37.616600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:37.617472 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:37.617481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:37.618167 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:37.618173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:37.618337 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:37.618919 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:37.618935 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:37.621542 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":306,"threads_started":1}
W20260620 23:22:37.622076 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:37.613848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43419 (local address 127.0.97.62:45519)
0620 23:22:37.614045 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:37.614051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:37.614370 (+ 319us) server_negotiation.cc:407] Connection header received
0620 23:22:37.614541 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:37.614546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:37.614651 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:37.614789 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:37.615700 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:37.616464 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:37.617613 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:37.618022 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:37.618165 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:37.621732 (+ 3567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43419: BlockingRecv error: recv got EOF from 127.0.97.2:43419 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 23:22:38 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:38 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:38.047888 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:38.040614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:34745)
0620 23:22:38.041048 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:38.041063 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:38.041171 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:38.041433 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:38.041436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:38.041779 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:38.041926 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:38.041934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:38.042931 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:38.042934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:38.043811 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:38.043821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:38.044426 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:38.044430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:38.044536 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:38.045078 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:38.045093 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:38.047661 (+ 2568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":213,"threads_started":1}
W20260620 23:22:38.048043 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:38.040701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34745 (local address 127.0.97.62:45519)
0620 23:22:38.040915 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:38.040921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:38.041136 (+ 215us) server_negotiation.cc:407] Connection header received
0620 23:22:38.041258 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:38.041262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:38.041317 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:38.041390 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:38.042032 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:38.042829 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:38.043923 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:38.044319 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:38.044400 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:38.047831 (+ 3431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34745: BlockingRecv error: recv got EOF from 127.0.97.3:34745 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 23:22:38 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:38 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:38.196775 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:38.188559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:37565)
0620 23:22:38.189086 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:38.189102 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:38.189208 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:38.189497 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:38.189500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:38.189892 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:38.190030 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:38.190037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:38.191382 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:38.191386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:38.192381 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:38.192394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:38.193271 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:38.193276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:38.193399 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:38.193973 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:38.193989 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:38.196576 (+ 2587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":305,"threads_started":1}
W20260620 23:22:38.196992 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:38.188770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37565 (local address 127.0.97.62:45519)
0620 23:22:38.188964 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:38.188968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:38.189175 (+ 207us) server_negotiation.cc:407] Connection header received
0620 23:22:38.189315 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:38.189318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:38.189374 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:38.189453 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:38.190174 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:38.191242 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:38.192508 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:38.193127 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:38.193235 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:38.196744 (+ 3509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37565: BlockingRecv error: recv got EOF from 127.0.97.1:37565 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 23:22:38 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:38 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:38.630807 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:38.622843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:55057)
0620 23:22:38.623373 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:38.623388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:38.623492 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:38.623765 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:38.623768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:38.624133 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:38.624269 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:38.624277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:38.625153 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:38.625157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:38.626136 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:38.626152 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:38.627146 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:38.627153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:38.627308 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:38.627807 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:38.627823 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:38.630455 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":296,"threads_started":1}
W20260620 23:22:38.630882 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:38.622885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55057 (local address 127.0.97.62:45519)
0620 23:22:38.623031 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:38.623035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:38.623454 (+ 419us) server_negotiation.cc:407] Connection header received
0620 23:22:38.623578 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:38.623582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:38.623647 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:38.623723 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:38.624378 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:38.625032 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:38.626279 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:38.626958 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:38.627072 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:38.630656 (+ 3584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55057: BlockingRecv error: recv got EOF from 127.0.97.2:55057 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 23:22:39 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:39 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:39.056528 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:39.048813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:51343)
0620 23:22:39.049535 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:39.049574 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:39.049707 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:39.050053 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:39.050056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:39.050588 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:39.050806 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:39.050815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:39.051751 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:39.051755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:39.052590 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:39.052601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:39.053185 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:39.053188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:39.053304 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:39.053834 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:39.053849 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:39.056320 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":493,"threads_started":1}
W20260620 23:22:39.056702 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:39.048930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51343 (local address 127.0.97.62:45519)
0620 23:22:39.049128 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:39.049132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:39.049673 (+ 541us) server_negotiation.cc:407] Connection header received
0620 23:22:39.049827 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:39.049831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:39.049888 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:39.049987 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:39.050931 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:39.051639 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:39.052706 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:39.053065 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:39.053164 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:39.056476 (+ 3312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51343: BlockingRecv error: recv got EOF from 127.0.97.3:51343 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 23:22:39 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:39 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:39.205468 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:39.197638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:47859)
0620 23:22:39.198212 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:39.198228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:39.198337 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:39.198600 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:39.198603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:39.199015 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:39.199181 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:39.199189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:39.200130 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:39.200134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:39.201037 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:39.201048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:39.201635 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:39.201639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:39.201736 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:39.202370 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:39.202410 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:39.205243 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":286,"threads_started":1}
W20260620 23:22:39.205549 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:39.197763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47859 (local address 127.0.97.62:45519)
0620 23:22:39.197939 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:39.197943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:39.198300 (+ 357us) server_negotiation.cc:407] Connection header received
0620 23:22:39.198425 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:39.198428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:39.198482 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:39.198558 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:39.199299 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:39.199999 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:39.201150 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:39.201521 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:39.201604 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:39.205395 (+ 3791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47859: BlockingRecv error: recv got EOF from 127.0.97.1:47859 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 23:22:39 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:39 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:39.639593 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:39.631696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:33399)
0620 23:22:39.632217 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:39.632232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:39.632336 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:39.632610 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:39.632620 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:39.632968 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:39.633134 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:39.633142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:39.634093 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:39.634096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:39.635043 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:39.635053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:39.635691 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:39.635695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:39.635796 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:39.636375 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:39.636391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:39.639289 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":248,"threads_started":1}
W20260620 23:22:39.639700 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:39.631742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33399 (local address 127.0.97.62:45519)
0620 23:22:39.631980 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:39.631990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:39.632303 (+ 313us) server_negotiation.cc:407] Connection header received
0620 23:22:39.632424 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:39.632427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:39.632495 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:39.632571 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:39.633263 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:39.633974 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:39.635160 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:39.635568 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:39.635665 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:39.639500 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33399: BlockingRecv error: recv got EOF from 127.0.97.2:33399 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 20 23:22:40 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:40 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:40.069727 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:40.057602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:53387)
0620 23:22:40.058346 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:40.058361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:40.058501 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:40.058891 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:40.058896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:40.059403 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:40.059562 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:40.059571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:40.060497 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:40.060500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:40.061355 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:40.061364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:40.061953 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:40.061956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:40.062056 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:40.066547 (+ 4491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:40.066565 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:40.069344 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":395,"threads_started":1}
W20260620 23:22:40.069820 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:40.057742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53387 (local address 127.0.97.62:45519)
0620 23:22:40.057944 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:40.057952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:40.058449 (+ 497us) server_negotiation.cc:407] Connection header received
0620 23:22:40.058633 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:40.058638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:40.058738 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:40.058834 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:40.059695 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:40.060386 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:40.061464 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:40.061838 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:40.061921 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:40.069563 (+ 7642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53387: BlockingRecv error: recv got EOF from 127.0.97.3:53387 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 23:22:40 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:40 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:40.214603 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:40.206410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:50997)
0620 23:22:40.206910 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:40.206926 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:40.207030 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:40.207293 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:40.207296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:40.207669 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:40.207808 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:40.207815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:40.208732 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:40.208735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:40.209654 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:40.209664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:40.210393 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:40.210399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:40.210544 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:40.211261 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:40.211285 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:40.214395 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":304,"threads_started":1}
W20260620 23:22:40.214846 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:40.206521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50997 (local address 127.0.97.62:45519)
0620 23:22:40.206767 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:40.206773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:40.206998 (+ 225us) server_negotiation.cc:407] Connection header received
0620 23:22:40.207115 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:40.207119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:40.207178 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:40.207253 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:40.207914 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:40.208617 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:40.209761 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:40.210228 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:40.210315 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:40.214586 (+ 4271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50997: BlockingRecv error: recv got EOF from 127.0.97.1:50997 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 23:22:40 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:40 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:40.648392 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:40.640485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:36311)
0620 23:22:40.640923 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:40.640938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:40.641045 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:40.641315 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:40.641317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:40.641687 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:40.641830 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:40.641837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:40.642942 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:40.642946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:40.643828 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:40.643839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:40.644449 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:40.644453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:40.644560 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:40.645095 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:40.645110 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:40.648057 (+ 2947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":251,"threads_started":1}
W20260620 23:22:40.648465 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:40.640582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36311 (local address 127.0.97.62:45519)
0620 23:22:40.640831 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:40.640836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:40.641015 (+ 179us) server_negotiation.cc:407] Connection header received
0620 23:22:40.641134 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:40.641137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:40.641206 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:40.641283 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:40.641937 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:40.642773 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:40.643938 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:40.644319 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:40.644403 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:40.648258 (+ 3855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36311: BlockingRecv error: recv got EOF from 127.0.97.2:36311 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 23:22:41 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:41 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:41.078474 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:41.070540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:58405)
0620 23:22:41.071026 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:41.071042 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:41.071147 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:41.071428 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:41.071430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:41.071819 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:41.071957 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:41.071964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:41.072823 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:41.072826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:41.073679 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:41.073701 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:41.074321 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:41.074325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:41.074434 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:41.074984 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:41.075000 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:41.078148 (+ 3148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":271,"threads_started":1}
W20260620 23:22:41.078742 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:41.070645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58405 (local address 127.0.97.62:45519)
0620 23:22:41.070837 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:41.070840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:41.071111 (+ 271us) server_negotiation.cc:407] Connection header received
0620 23:22:41.071238 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:41.071242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:41.071307 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:41.071384 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:41.072071 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:41.072710 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:41.073802 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:41.074186 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:41.074271 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:41.078417 (+ 4146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58405: BlockingRecv error: recv got EOF from 127.0.97.3:58405 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 23:22:41 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:41 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:41.223976 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:41.215719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:45309)
0620 23:22:41.216196 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:41.216211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:41.216377 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:41.216705 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:41.216710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:41.217088 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:41.217240 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:41.217247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:41.218245 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:41.218252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:41.219418 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:41.219431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:41.220247 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:41.220253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:41.220436 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:41.221099 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:41.221117 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:41.223754 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":288,"threads_started":1}
W20260620 23:22:41.224120 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:41.215757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45309 (local address 127.0.97.62:45519)
0620 23:22:41.215962 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:41.215969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:41.216299 (+ 330us) server_negotiation.cc:407] Connection header received
0620 23:22:41.216497 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:41.216504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:41.216577 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:41.216676 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:41.217381 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:41.218087 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:41.219585 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:41.220121 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:41.220220 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:41.223914 (+ 3694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45309: BlockingRecv error: recv got EOF from 127.0.97.1:45309 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 23:22:41 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:41 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:41.657524 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:41.649477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:39059)
0620 23:22:41.650028 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:41.650051 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:41.650162 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:41.650451 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:41.650454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:41.650928 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:41.651106 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:41.651114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:41.651975 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:41.651978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:41.652890 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:41.652900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:41.653724 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:41.653733 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:41.653885 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:41.654566 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:41.654590 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:41.657264 (+ 2674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":319,"threads_started":1}
W20260620 23:22:41.657670 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:41.649530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39059 (local address 127.0.97.62:45519)
0620 23:22:41.649694 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:41.649698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:41.650125 (+ 427us) server_negotiation.cc:407] Connection header received
0620 23:22:41.650251 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:41.650254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:41.650312 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:41.650389 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:41.651218 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:41.651869 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:41.652998 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:41.653540 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:41.653628 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:41.657441 (+ 3813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39059: BlockingRecv error: recv got EOF from 127.0.97.2:39059 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 23:22:42 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:42 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:42.089366 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:42.080182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:36307)
0620 23:22:42.080817 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:42.080831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:42.080978 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:42.081350 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:42.081354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:42.081846 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:42.082100 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:42.082110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:42.083471 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:42.083478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:42.084394 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:42.084405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:42.085113 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:42.085119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:42.085301 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:42.085902 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:42.085919 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:42.089011 (+ 3092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":391,"threads_started":1}
W20260620 23:22:42.089402 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:42.080239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36307 (local address 127.0.97.62:45519)
0620 23:22:42.080520 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:42.080529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:42.080915 (+ 386us) server_negotiation.cc:407] Connection header received
0620 23:22:42.081114 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:42.081120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:42.081220 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:42.081323 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:42.082310 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:42.083346 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:42.084548 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:42.084963 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:42.085097 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:42.089202 (+ 4105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36307: BlockingRecv error: recv got EOF from 127.0.97.3:36307 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Jun 20 23:22:42 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:42 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:42.232825 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:42.224961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:54297)
0620 23:22:42.225378 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:42.225393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:42.225494 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:42.225753 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:42.225756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:42.226071 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:42.226211 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:42.226219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:42.227156 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:42.227159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:42.228341 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:42.228355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:42.229316 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:42.229328 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:42.229439 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:42.230001 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:42.230017 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:42.232617 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":247,"threads_started":1}
W20260620 23:22:42.233004 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:42.225095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54297 (local address 127.0.97.62:45519)
0620 23:22:42.225305 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:42.225311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:42.225465 (+ 154us) server_negotiation.cc:407] Connection header received
0620 23:22:42.225581 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:42.225585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:42.225641 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:42.225740 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:42.226323 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:42.227031 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:42.228480 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:42.229172 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:42.229264 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:42.232791 (+ 3527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54297: BlockingRecv error: recv got EOF from 127.0.97.1:54297 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 23:22:42 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:42 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:42.667691 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:42.658682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:42441)
0620 23:22:42.659182 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:42.659200 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:42.659319 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:42.659640 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:42.659644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:42.660086 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:42.660253 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:42.660263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:42.661464 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:42.661469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:42.663045 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:42.663059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:42.663745 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:42.663751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:42.663921 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:42.664510 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:42.664533 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:42.667487 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":258,"threads_started":1}
W20260620 23:22:42.667829 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:42.658787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42441 (local address 127.0.97.62:45519)
0620 23:22:42.658988 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:42.658995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:42.659290 (+ 295us) server_negotiation.cc:407] Connection header received
0620 23:22:42.659426 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:42.659430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:42.659485 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:42.659579 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:42.660377 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:42.661329 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:42.663183 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:42.663601 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:42.663708 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:42.667640 (+ 3932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42441: BlockingRecv error: recv got EOF from 127.0.97.2:42441 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 23:22:43 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:43 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:43.098445 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:43.090379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:40065)
0620 23:22:43.091069 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:43.091090 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:43.091237 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:43.091559 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:43.091564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:43.091949 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:43.092080 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:43.092088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:43.093060 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:43.093065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:43.094003 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:43.094014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:43.094800 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:43.094806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:43.094976 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:43.095511 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:43.095527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:43.098130 (+ 2603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":467,"threads_started":1}
W20260620 23:22:43.098536 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:43.090535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40065 (local address 127.0.97.62:45519)
0620 23:22:43.090736 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:43.090742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:43.091178 (+ 436us) server_negotiation.cc:407] Connection header received
0620 23:22:43.091360 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:43.091366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:43.091436 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:43.091531 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:43.092225 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:43.092928 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:43.094151 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:43.094611 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:43.094788 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:43.098314 (+ 3526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40065: BlockingRecv error: recv got EOF from 127.0.97.3:40065 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 23:22:43 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:43 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:43.241762 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:43.233747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:50189)
0620 23:22:43.234219 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:43.234234 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:43.234379 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:43.234726 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:43.234731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:43.235128 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:43.235312 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:43.235320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:43.236292 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:43.236299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:43.237184 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:43.237195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:43.237862 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:43.237868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:43.238036 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:43.238648 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:43.238664 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:43.241437 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":256,"threads_started":1}
W20260620 23:22:43.241962 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:43.233802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50189 (local address 127.0.97.62:45519)
0620 23:22:43.234004 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:43.234011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:43.234313 (+ 302us) server_negotiation.cc:407] Connection header received
0620 23:22:43.234501 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:43.234506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:43.234571 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:43.234701 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:43.235454 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:43.236168 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:43.237318 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:43.237721 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:43.237851 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:43.241608 (+ 3757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50189: BlockingRecv error: recv got EOF from 127.0.97.1:50189 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 23:22:43 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:43 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:43.676931 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:43.668692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:51707)
0620 23:22:43.669191 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:43.669206 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:43.669350 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:43.669673 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:43.669677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:43.670108 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:43.670296 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:43.670303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:43.671345 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:43.671353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:43.672299 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:43.672310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:43.673112 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:43.673117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:43.673298 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:43.673837 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:43.673853 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:43.676608 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":267,"threads_started":1}
W20260620 23:22:43.677328 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:43.668773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51707 (local address 127.0.97.62:45519)
0620 23:22:43.669005 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:43.669012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:43.669289 (+ 277us) server_negotiation.cc:407] Connection header received
0620 23:22:43.669476 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:43.669483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:43.669548 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:43.669655 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:43.670433 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:43.671222 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:43.672452 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:43.672971 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:43.673099 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:43.676848 (+ 3749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51707: BlockingRecv error: recv got EOF from 127.0.97.2:51707 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 23:22:44 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:44 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:44.107184 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:44.099421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:53725)
0620 23:22:44.100092 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:44.100108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:44.100213 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:44.100492 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:44.100495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:44.100875 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:44.101044 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:44.101052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:44.101930 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:44.101933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:44.102861 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:44.102872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:44.103517 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:44.103520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:44.103632 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:44.104252 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:44.104269 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:44.106936 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":442,"threads_started":1}
W20260620 23:22:44.107326 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:44.099616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53725 (local address 127.0.97.62:45519)
0620 23:22:44.099823 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:44.099829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:44.100168 (+ 339us) server_negotiation.cc:407] Connection header received
0620 23:22:44.100310 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:44.100314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:44.100371 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:44.100449 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:44.101159 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:44.101831 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:44.102973 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:44.103399 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:44.103487 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:44.107117 (+ 3630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53725: BlockingRecv error: recv got EOF from 127.0.97.3:53725 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 23:22:44 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:44 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:44.250480 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:44.242593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:41895)
0620 23:22:44.243092 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:44.243107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:44.243265 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:44.243584 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:44.243589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:44.243979 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:44.244121 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:44.244129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:44.245189 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:44.245194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:44.246099 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:44.246109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:44.246871 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:44.246876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:44.247062 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:44.247605 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:44.247622 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:44.250257 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":285,"threads_started":1}
W20260620 23:22:44.250838 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:44.242648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41895 (local address 127.0.97.62:45519)
0620 23:22:44.242884 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:44.242892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:44.243196 (+ 304us) server_negotiation.cc:407] Connection header received
0620 23:22:44.243383 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:44.243388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:44.243456 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:44.243553 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:44.244269 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:44.245052 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:44.246239 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:44.246659 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:44.246865 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:44.250442 (+ 3577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41895: BlockingRecv error: recv got EOF from 127.0.97.1:41895 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 23:22:44 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:44 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:44.685662 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:44.677807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:35327)
0620 23:22:44.678293 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:44.678307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:44.678454 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:44.678803 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:44.678807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:44.679247 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:44.679427 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:44.679435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:44.680400 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:44.680406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:44.681339 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:44.681350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:44.682008 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:44.682013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:44.682206 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:44.682759 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:44.682775 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:44.685352 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":250,"threads_started":1}
W20260620 23:22:44.685736 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:44.677871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35327 (local address 127.0.97.62:45519)
0620 23:22:44.678085 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:44.678092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:44.678389 (+ 297us) server_negotiation.cc:407] Connection header received
0620 23:22:44.678570 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:44.678575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:44.678641 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:44.678776 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:44.679570 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:44.680263 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:44.681474 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:44.681872 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:44.681997 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:44.685533 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35327: BlockingRecv error: recv got EOF from 127.0.97.2:35327 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 23:22:45 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:45 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:45.119076 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:45.108064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:47047)
0620 23:22:45.108592 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:45.108610 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:45.108745 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:45.109043 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:45.109048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:45.109520 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:45.109737 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:45.109749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:45.110725 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:45.110730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:45.111748 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:45.111759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:45.112365 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:45.112369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:45.112484 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:45.115650 (+ 3166us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:45.115682 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:45.118750 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":271,"threads_started":1}
W20260620 23:22:45.119181 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:45.108131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47047 (local address 127.0.97.62:45519)
0620 23:22:45.108372 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:45.108379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:45.108689 (+ 310us) server_negotiation.cc:407] Connection header received
0620 23:22:45.108845 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:45.108849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:45.108906 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:45.108980 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:45.109864 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:45.110554 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:45.111869 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:45.112255 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:45.112347 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:45.118942 (+ 6595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47047: BlockingRecv error: recv got EOF from 127.0.97.3:47047 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 23:22:45 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:45 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:45.259213 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:45.251485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:36337)
0620 23:22:45.251954 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:45.251970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:45.252079 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:45.252360 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:45.252363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:45.252726 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:45.252918 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:45.252926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:45.253932 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:45.253935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:45.254912 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:45.254930 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:45.255636 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:45.255640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:45.255753 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:45.256283 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:45.256299 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:45.258999 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":256,"threads_started":1}
W20260620 23:22:45.259352 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:45.251638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36337 (local address 127.0.97.62:45519)
0620 23:22:45.251856 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:45.251861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:45.252048 (+ 187us) server_negotiation.cc:407] Connection header received
0620 23:22:45.252172 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:45.252175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:45.252234 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:45.252312 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:45.253044 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:45.253807 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:45.255068 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:45.255492 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:45.255584 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:45.259164 (+ 3580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36337: BlockingRecv error: recv got EOF from 127.0.97.1:36337 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 23:22:45 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:45 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:45.694505 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:45.686429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:38843)
0620 23:22:45.686976 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:45.686995 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:45.687144 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:45.687485 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:45.687489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:45.687911 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:45.688076 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:45.688085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:45.689038 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:45.689045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:45.689921 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:45.689933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:45.690747 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:45.690753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:45.690915 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:45.691432 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:45.691447 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:45.694295 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":293,"threads_started":1}
W20260620 23:22:45.694655 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:45.686579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38843 (local address 127.0.97.62:45519)
0620 23:22:45.686799 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:45.686807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:45.687080 (+ 273us) server_negotiation.cc:407] Connection header received
0620 23:22:45.687282 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:45.687287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:45.687362 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:45.687457 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:45.688224 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:45.688906 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:45.690081 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:45.690535 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:45.690674 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:45.694449 (+ 3775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38843: BlockingRecv error: recv got EOF from 127.0.97.2:38843 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 23:22:46 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:46 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:46.127301 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:46.119913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:43885)
0620 23:22:46.120422 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:46.120438 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:46.120541 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:46.120803 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:46.120806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:46.121156 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:46.121323 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:46.121330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:46.122238 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:46.122241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:46.123165 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:46.123175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:46.123785 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:46.123789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:46.123894 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:46.124508 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:46.124524 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:46.127066 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":269,"threads_started":1}
W20260620 23:22:46.127560 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:46.119940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43885 (local address 127.0.97.62:45519)
0620 23:22:46.120110 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:46.120114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:46.120503 (+ 389us) server_negotiation.cc:407] Connection header received
0620 23:22:46.120628 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:46.120631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:46.120688 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:46.120763 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:46.121431 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:46.122121 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:46.123272 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:46.123645 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:46.123729 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:46.127257 (+ 3528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43885: BlockingRecv error: recv got EOF from 127.0.97.3:43885 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 23:22:46 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:46 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:46.268080 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:46.260168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:54027)
0620 23:22:46.260644 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:46.260659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:46.260818 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:46.261143 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:46.261147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:46.261546 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:46.261694 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:46.261702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:46.262743 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:46.262748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:46.263640 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:46.263650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:46.264345 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:46.264350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:46.264526 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:46.265142 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:46.265158 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:46.267844 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":250,"threads_started":1}
W20260620 23:22:46.268368 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:46.260238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54027 (local address 127.0.97.62:45519)
0620 23:22:46.260533 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:46.260538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:46.260753 (+ 215us) server_negotiation.cc:407] Connection header received
0620 23:22:46.260941 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:46.260947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:46.261014 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:46.261116 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:46.261837 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:46.262568 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:46.263789 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:46.264200 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:46.264334 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:46.268067 (+ 3733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54027: BlockingRecv error: recv got EOF from 127.0.97.1:54027 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Jun 20 23:22:46 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:46 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:46.702961 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:46.695490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:50967)
0620 23:22:46.695967 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:46.695988 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:46.696118 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:46.696415 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:46.696418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:46.696826 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:46.696980 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:46.696988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:46.697914 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:46.697918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:46.698822 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:46.698833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:46.699449 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:46.699453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:46.699562 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:46.700116 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:46.700132 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:46.702655 (+ 2523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":250,"threads_started":1}
W20260620 23:22:46.703023 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:46.695625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50967 (local address 127.0.97.62:45519)
0620 23:22:46.695773 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:46.695777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:46.696070 (+ 293us) server_negotiation.cc:407] Connection header received
0620 23:22:46.696219 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:46.696223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:46.696291 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:46.696373 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:46.697110 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:46.697798 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:46.698937 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:46.699340 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:46.699427 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:46.702847 (+ 3420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50967: BlockingRecv error: recv got EOF from 127.0.97.2:50967 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 23:22:47 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:47 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:47.136264 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:47.128195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.3:52327)
0620 23:22:47.128651 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:47.128666 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:47.128827 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:47.129134 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:47.129139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:47.129482 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:47.129663 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:47.129674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.130951 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.130959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:47.132097 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:47.132109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.132752 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.132756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:47.132870 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:47.133502 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:47.133519 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:47.136020 (+ 2501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":247,"threads_started":1}
W20260620 23:22:47.136444 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:47.128268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52327 (local address 127.0.97.62:45519)
0620 23:22:47.128493 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:47.128500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:47.128782 (+ 282us) server_negotiation.cc:407] Connection header received
0620 23:22:47.128943 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:47.128947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:47.129007 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:47.129114 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:47.129791 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.130801 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.132213 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.132631 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.132729 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:47.136183 (+ 3454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52327: BlockingRecv error: recv got EOF from 127.0.97.3:52327 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 23:22:47 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:47 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:47.277112 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:47.269120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.1:60255)
0620 23:22:47.269589 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:47.269603 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:47.269751 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:47.270164 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:47.270172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:47.270749 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:47.270997 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:47.271009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.272084 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.272088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:47.272949 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:47.272959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.273563 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.273567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:47.273673 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:47.274296 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:47.274313 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:47.276881 (+ 2568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":260,"threads_started":1}
W20260620 23:22:47.277266 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:47.269227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60255 (local address 127.0.97.62:45519)
0620 23:22:47.269433 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:47.269440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:47.269685 (+ 245us) server_negotiation.cc:407] Connection header received
0620 23:22:47.269898 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:47.269906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:47.269999 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:47.270103 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:47.271162 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.271969 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.273061 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.273455 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.273540 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:47.277047 (+ 3507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60255: BlockingRecv error: recv got EOF from 127.0.97.1:60255 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 23:22:47 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:22:47 dist-test-slave-5kzz krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:22:47.711967 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:47.704114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45519 (local address 127.0.97.2:60969)
0620 23:22:47.704640 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:47.704656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:47.704767 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:47.705116 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:47.705120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:47.705528 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:47.705710 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:47.705718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.706765 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.706770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:47.707659 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:47.707668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.708365 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.708371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:47.708540 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:47.709079 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:47.709095 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:47.711665 (+ 2570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45519: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":318,"threads_started":1}
W20260620 23:22:47.712121 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:47.704249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60969 (local address 127.0.97.62:45519)
0620 23:22:47.704416 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:47.704420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:47.704730 (+ 310us) server_negotiation.cc:407] Connection header received
0620 23:22:47.704914 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:47.704919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:47.704978 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:47.705088 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:47.705851 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.706582 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.707803 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.708207 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.708328 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:47.711884 (+ 3556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60969: BlockingRecv error: recv got EOF from 127.0.97.2:60969 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:22:47.972878 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 9214
I20260620 23:22:47.984633 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:47.973902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.0.97.1:34715)
0620 23:22:47.974444 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:47.974451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:47.974475 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:22:47.974529 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:47.974535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:47.974835 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:47.974990 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:47.976270 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.977326 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.978428 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:47.979086 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:47.979200 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:47.981660 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:47.981668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:47.981682 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:47.981719 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:47.984120 (+ 2401us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:22:47.984359 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47374: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":285,"threads_started":1}
W20260620 23:22:47.984743 9674 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47374: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:22:47.984807 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 9214: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:34715: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:22:47.985018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 9214
I20260620 23:22:48.001423 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 9365
I20260620 23:22:48.013502 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:48.002439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.0.97.2:45229)
0620 23:22:48.003029 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:48.003036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:48.003064 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:22:48.003119 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:48.003124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:48.003352 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:48.003564 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:48.004803 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:48.005838 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:48.007273 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:48.007820 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:48.007968 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:48.010359 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:48.010368 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:48.010381 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:48.010417 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:48.012883 (+ 2466us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:22:48.013174 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36040: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":321,"threads_started":1}
W20260620 23:22:48.013510 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 9365: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:45229: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:22:48.013607 9675 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36040: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:22:48.013660 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 9365
I20260620 23:22:48.030488 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 9496
I20260620 23:22:48.041874 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:48.031483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.0.97.3:39709)
0620 23:22:48.032023 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:48.032029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:48.032054 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:22:48.032108 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:48.032114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:48.032320 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:48.032487 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:48.033678 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:48.034805 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:48.035863 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:48.036393 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:48.036483 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:48.038831 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:48.038857 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:48.038870 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:48.038908 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:48.041296 (+ 2388us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:22:48.041585 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59116: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":288,"threads_started":1}
W20260620 23:22:48.041920 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 9496: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:39709: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:22:48.041975 9676 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59116: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:22:48.042107 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 9496
I20260620 23:22:48.057928 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 9145
I20260620 23:22:48.059178 9207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:48.170289 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 9145
2026-06-20T23:22:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20704 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T23:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:48Z Disabled control of system clock
I20260620 23:22:48.243465 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37631
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:22:48.552942 9681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:48.553237 9681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:48.553313 9681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:48.560272 9681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:48.560382 9681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:48.560420 9681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:48.560467 9681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:48.560511 9681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:22:48.569380 9681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:48.570721 9681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:48.572481 9681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:48.584174 9686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:48.584266 9689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:48.584159 9687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:48.584790 9681 server_base.cc:1034] running on GCE node
I20260620 23:22:48.585655 9681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:48.587034 9681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:48.588248 9681 hybrid_clock.cc:648] HybridClock initialized: now 1781997768588189 us; error 53 us; skew 500 ppm
I20260620 23:22:48.588567 9681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:48.590909 9681 webserver.cc:460] Webserver started at http://127.0.97.62:42601/ using document root <none> and password file <none>
I20260620 23:22:48.591585 9681 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:48.591708 9681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:48.592007 9681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:48.594790 9681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "f1f555ed76874422bec2db438f1270cc"
format_stamp: "Formatted at 2026-06-20 23:22:48 on dist-test-slave-5kzz"
server_key: "8f1811d26e5040d27c2fe7448da5f4a3"
server_key_iv: "547e52f53f840cd72b0fa38851e8d225"
server_key_version: "encryptionkey@0"
I20260620 23:22:48.595593 9681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "f1f555ed76874422bec2db438f1270cc"
format_stamp: "Formatted at 2026-06-20 23:22:48 on dist-test-slave-5kzz"
server_key: "8f1811d26e5040d27c2fe7448da5f4a3"
server_key_iv: "547e52f53f840cd72b0fa38851e8d225"
server_key_version: "encryptionkey@0"
I20260620 23:22:48.600422 9681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 23:22:48.603619 9695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:48.604794 9681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 23:22:48.604964 9681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "f1f555ed76874422bec2db438f1270cc"
format_stamp: "Formatted at 2026-06-20 23:22:48 on dist-test-slave-5kzz"
server_key: "8f1811d26e5040d27c2fe7448da5f4a3"
server_key_iv: "547e52f53f840cd72b0fa38851e8d225"
server_key_version: "encryptionkey@0"
I20260620 23:22:48.605158 9681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:48.620833 9681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:48.621562 9681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:48.621833 9681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:48.639480 9681 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37631
I20260620 23:22:48.639497 9746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37631 every 8 connection(s)
I20260620 23:22:48.640717 9681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:48.642417 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 9681
I20260620 23:22:48.642618 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:22:48.643309 388 external_mini_cluster.cc:1428] Setting key a5323bf8447a6af85605cd6ea78fde89
I20260620 23:22:48.644501 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:48.657575 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:48.645219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.0.97.62:37631)
0620 23:22:48.646211 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:48.646224 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:48.646269 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:22:48.647976 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:48.648003 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:48.648573 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:48.648976 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:48.649499 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:48.651483 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:48.652731 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:48.653682 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:48.653870 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:48.654941 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:48.654955 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:48.654972 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:48.655668 (+ 696us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:48.655750 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:48.655778 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:48.655907 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:48.656099 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 23:22:48.656405 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":315,"threads_started":1}
I20260620 23:22:48.659003 9747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc: Bootstrap starting.
I20260620 23:22:48.662014 9747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:48.663002 9747 log.cc:824] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:48.665369 9747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc: No bootstrap required, opened a new log
I20260620 23:22:48.672138 9747 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f555ed76874422bec2db438f1270cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37631 } }
I20260620 23:22:48.672446 9747 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:48.672560 9747 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f555ed76874422bec2db438f1270cc, State: Initialized, Role: FOLLOWER
I20260620 23:22:48.673113 9747 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f555ed76874422bec2db438f1270cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37631 } }
I20260620 23:22:48.673337 9747 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:48.673455 9747 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:48.673666 9747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:48.675681 9747 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f555ed76874422bec2db438f1270cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37631 } }
I20260620 23:22:48.676158 9747 leader_election.cc:302] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1f555ed76874422bec2db438f1270cc; no voters:
I20260620 23:22:48.676719 9747 leader_election.cc:288] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:22:48.676908 9752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:48.677789 9752 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [term 1 LEADER]: Becoming Leader. State: Replica: f1f555ed76874422bec2db438f1270cc, State: Running, Role: LEADER
I20260620 23:22:48.678243 9752 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f555ed76874422bec2db438f1270cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37631 } }
I20260620 23:22:48.678840 9747 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:48.681041 9753 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1f555ed76874422bec2db438f1270cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f555ed76874422bec2db438f1270cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37631 } } }
I20260620 23:22:48.681238 9754 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1f555ed76874422bec2db438f1270cc. Latest consensus state: current_term: 1 leader_uuid: "f1f555ed76874422bec2db438f1270cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f555ed76874422bec2db438f1270cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37631 } } }
I20260620 23:22:48.681432 9753 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:48.681495 9754 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:48.683583 9761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:48.688486 9761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:48.696947 9761 catalog_manager.cc:1347] Generated new cluster ID: 6b68e7741a884bd997f6d16a5dc09ee8
I20260620 23:22:48.697042 9761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:48.734221 9761 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:22:48.736001 9761 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:48.744032 9761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1f555ed76874422bec2db438f1270cc: Generated new TSK 0
I20260620 23:22:48.744982 9761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:22:48.754964 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37631
--builtin_ntp_servers=127.0.97.20:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 23:22:49.070216 9771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:49.070500 9771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:49.070640 9771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:49.077749 9771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:49.077840 9771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:49.078003 9771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:22:49.087211 9771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:49.088444 9771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:49.090085 9771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:49.102094 9779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:49.102156 9777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:49.102346 9776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:49.103186 9771 server_base.cc:1034] running on GCE node
I20260620 23:22:49.103679 9771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:49.104565 9771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:49.105790 9771 hybrid_clock.cc:648] HybridClock initialized: now 1781997769105730 us; error 55 us; skew 500 ppm
I20260620 23:22:49.106212 9771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:49.108695 9771 webserver.cc:460] Webserver started at http://127.0.97.1:36691/ using document root <none> and password file <none>
I20260620 23:22:49.109338 9771 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:49.109418 9771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:49.109686 9771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:49.112288 9771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "78cc734bd3794d1f8f145b4f4e093014"
format_stamp: "Formatted at 2026-06-20 23:22:49 on dist-test-slave-5kzz"
server_key: "e9312ed091c1b8d51a16052f9f839c2a"
server_key_iv: "3ead9344135f9516bf4edbe3bdc742f2"
server_key_version: "encryptionkey@0"
I20260620 23:22:49.112983 9771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "78cc734bd3794d1f8f145b4f4e093014"
format_stamp: "Formatted at 2026-06-20 23:22:49 on dist-test-slave-5kzz"
server_key: "e9312ed091c1b8d51a16052f9f839c2a"
server_key_iv: "3ead9344135f9516bf4edbe3bdc742f2"
server_key_version: "encryptionkey@0"
I20260620 23:22:49.117654 9771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 23:22:49.121796 9785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:49.123178 9771 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 23:22:49.123313 9771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "78cc734bd3794d1f8f145b4f4e093014"
format_stamp: "Formatted at 2026-06-20 23:22:49 on dist-test-slave-5kzz"
server_key: "e9312ed091c1b8d51a16052f9f839c2a"
server_key_iv: "3ead9344135f9516bf4edbe3bdc742f2"
server_key_version: "encryptionkey@0"
I20260620 23:22:49.123468 9771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:49.142553 9771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:49.143397 9771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:49.143605 9771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:49.144433 9771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:49.145769 9771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:49.145853 9771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:49.145925 9771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:49.145977 9771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:49.176909 9771 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37833
I20260620 23:22:49.176936 9897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37833 every 8 connection(s)
I20260620 23:22:49.178143 9771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:22:49.188227 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 9771
I20260620 23:22:49.188408 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:22:49.189213 388 external_mini_cluster.cc:1428] Setting key c31b04fabbeb92ff303c2f05b5a9b600
I20260620 23:22:49.190950 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:49.180716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53231 (local address 127.0.97.62:37631)
0620 23:22:49.181137 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:49.181143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:49.182414 (+ 1271us) server_negotiation.cc:407] Connection header received
0620 23:22:49.184115 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:49.184118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:49.184177 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:49.184272 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:49.185264 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:49.185973 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:49.187990 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:49.188642 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:49.188781 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:49.190083 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:49.190088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:49.190093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:49.190118 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:49.190155 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:49.190185 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:49.190278 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:49.190493 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 23:22:49.190580 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":202,"threads_started":1}
I20260620 23:22:49.191895 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37631
--builtin_ntp_servers=127.0.97.20:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:22:49.191932 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:49.181363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37631 (local address 127.0.97.1:53231)
0620 23:22:49.182248 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:49.182283 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:49.183890 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:49.184425 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:49.184431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:49.184509 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:49.185122 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:49.185138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:49.186120 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:49.186123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:49.187857 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:49.187869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:49.188848 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:49.188853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:49.189045 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:49.189777 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:49.189780 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:49.189836 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:49.189904 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:49.190287 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:49.190300 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:49.190309 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:22:49.190485 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 23:22:49.190872 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":254,"threads_started":1}
I20260620 23:22:49.193332 9898 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37631
I20260620 23:22:49.193665 9898 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:49.194530 9898 heartbeater.cc:511] Master 127.0.97.62:37631 requested a full tablet report, sending...
I20260620 23:22:49.196646 9712 ts_manager.cc:194] Registered new tserver with Master: 78cc734bd3794d1f8f145b4f4e093014 (127.0.97.1:37833)
I20260620 23:22:49.198717 9712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:53231
W20260620 23:22:49.503880 9902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:49.504128 9902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:49.504205 9902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:49.511215 9902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:49.511298 9902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:49.511394 9902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:22:49.519972 9902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:49.521147 9902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:49.522855 9902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:49.535269 9908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:49.535295 9907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:49.535756 9910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:49.536499 9902 server_base.cc:1034] running on GCE node
I20260620 23:22:49.537072 9902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:49.537981 9902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:49.539194 9902 hybrid_clock.cc:648] HybridClock initialized: now 1781997769539143 us; error 63 us; skew 500 ppm
I20260620 23:22:49.539505 9902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:49.541637 9902 webserver.cc:460] Webserver started at http://127.0.97.2:41499/ using document root <none> and password file <none>
I20260620 23:22:49.542321 9902 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:49.542451 9902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:49.542831 9902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:49.545550 9902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "b84bbdc08b2a4de9a589bfe8e9079da7"
format_stamp: "Formatted at 2026-06-20 23:22:49 on dist-test-slave-5kzz"
server_key: "21446b012dd3b23b6d389fd402c28e66"
server_key_iv: "4c793b74dfa1211224491d9debcf379b"
server_key_version: "encryptionkey@0"
I20260620 23:22:49.546307 9902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b84bbdc08b2a4de9a589bfe8e9079da7"
format_stamp: "Formatted at 2026-06-20 23:22:49 on dist-test-slave-5kzz"
server_key: "21446b012dd3b23b6d389fd402c28e66"
server_key_iv: "4c793b74dfa1211224491d9debcf379b"
server_key_version: "encryptionkey@0"
I20260620 23:22:49.551044 9902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 23:22:49.554242 9916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:49.555516 9902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:22:49.555688 9902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "b84bbdc08b2a4de9a589bfe8e9079da7"
format_stamp: "Formatted at 2026-06-20 23:22:49 on dist-test-slave-5kzz"
server_key: "21446b012dd3b23b6d389fd402c28e66"
server_key_iv: "4c793b74dfa1211224491d9debcf379b"
server_key_version: "encryptionkey@0"
I20260620 23:22:49.555917 9902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:49.574378 9902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:49.575428 9902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:49.575747 9902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:49.576805 9902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:49.578608 9902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:49.578737 9902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 23:22:49.578850 9902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:49.578940 9902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:49.610350 9902 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39535
I20260620 23:22:49.610415 10028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39535 every 8 connection(s)
I20260620 23:22:49.611642 9902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:22:49.615129 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 9902
I20260620 23:22:49.615301 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:22:49.615839 388 external_mini_cluster.cc:1428] Setting key 0b6e412b07f998114712b5fe28e8a44c
I20260620 23:22:49.618716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37631
--builtin_ntp_servers=127.0.97.20:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:22:49.625363 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:49.613997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52905 (local address 127.0.97.62:37631)
0620 23:22:49.614198 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:49.614204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:49.616012 (+ 1808us) server_negotiation.cc:407] Connection header received
0620 23:22:49.618075 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:49.618078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:49.618136 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:49.618214 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:49.619559 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:49.620341 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:49.622750 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:49.623108 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:49.623201 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:49.624705 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:49.624709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:49.624712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:49.624728 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:49.624757 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:49.624767 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:49.624832 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:49.625077 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 23:22:49.625142 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:22:49.626515 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:49.614612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37631 (local address 127.0.97.2:52905)
0620 23:22:49.615799 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:49.615860 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:49.617802 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:49.618473 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:49.618485 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:49.618601 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:49.619402 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:49.619428 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:49.620515 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:49.620523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:49.622565 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:49.622586 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:49.623320 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:49.623330 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:49.623645 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:49.624370 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:49.624376 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:49.624428 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:49.624521 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:49.624859 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:49.624872 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:49.624882 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:22:49.625062 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 23:22:49.625474 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":381,"threads_started":1}
I20260620 23:22:49.627925 10029 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37631
I20260620 23:22:49.628221 10029 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:49.628923 10029 heartbeater.cc:511] Master 127.0.97.62:37631 requested a full tablet report, sending...
I20260620 23:22:49.630231 9712 ts_manager.cc:194] Registered new tserver with Master: b84bbdc08b2a4de9a589bfe8e9079da7 (127.0.97.2:39535)
I20260620 23:22:49.631021 9712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:52905
W20260620 23:22:49.933362 10032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:49.933634 10032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:49.933804 10032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:49.940572 10032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:49.940693 10032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:49.940860 10032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:22:49.949395 10032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:49.950652 10032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:49.952325 10032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:49.964512 10037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:49.964485 10040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:49.964653 10038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:49.965258 10032 server_base.cc:1034] running on GCE node
I20260620 23:22:49.965853 10032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:49.966720 10032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:49.967969 10032 hybrid_clock.cc:648] HybridClock initialized: now 1781997769967903 us; error 56 us; skew 500 ppm
I20260620 23:22:49.968276 10032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:49.970300 10032 webserver.cc:460] Webserver started at http://127.0.97.3:36545/ using document root <none> and password file <none>
I20260620 23:22:49.971050 10032 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:49.971174 10032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:49.971495 10032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:49.974102 10032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "d7812929d04944aba5cf01f92f575dfc"
format_stamp: "Formatted at 2026-06-20 23:22:49 on dist-test-slave-5kzz"
server_key: "8c08ec8e2236897cc7d6ff0c73439a69"
server_key_iv: "1f0337a3e8214faced212b651aec0fff"
server_key_version: "encryptionkey@0"
I20260620 23:22:49.974908 10032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d7812929d04944aba5cf01f92f575dfc"
format_stamp: "Formatted at 2026-06-20 23:22:49 on dist-test-slave-5kzz"
server_key: "8c08ec8e2236897cc7d6ff0c73439a69"
server_key_iv: "1f0337a3e8214faced212b651aec0fff"
server_key_version: "encryptionkey@0"
I20260620 23:22:49.979614 10032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 23:22:49.983071 10046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:49.984310 10032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:22:49.984476 10032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "d7812929d04944aba5cf01f92f575dfc"
format_stamp: "Formatted at 2026-06-20 23:22:49 on dist-test-slave-5kzz"
server_key: "8c08ec8e2236897cc7d6ff0c73439a69"
server_key_iv: "1f0337a3e8214faced212b651aec0fff"
server_key_version: "encryptionkey@0"
I20260620 23:22:49.984675 10032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:50.002477 10032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:50.003302 10032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:50.003561 10032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:50.004426 10032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:50.005775 10032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:50.005892 10032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:50.006022 10032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:50.006105 10032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:50.038082 10032 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36177
I20260620 23:22:50.038130 10158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36177 every 8 connection(s)
I20260620 23:22:50.039453 10032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:22:50.042013 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 10032
I20260620 23:22:50.042168 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:22:50.042622 388 external_mini_cluster.cc:1428] Setting key a622c6a4081ca356edfcd5265969b043
I20260620 23:22:50.051064 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:50.041828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50429 (local address 127.0.97.62:37631)
0620 23:22:50.042035 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:50.042043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:50.043650 (+ 1607us) server_negotiation.cc:407] Connection header received
0620 23:22:50.045486 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:50.045490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:50.045549 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:50.045629 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:50.046556 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:50.047298 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:50.048734 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:50.049114 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:50.049201 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:50.050279 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:50.050283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:50.050286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:50.050301 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:50.050326 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:50.050336 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:50.050404 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:50.050610 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 23:22:50.050729 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:22:50.051997 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:50.042459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37631 (local address 127.0.97.3:50429)
0620 23:22:50.043488 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:50.043542 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:50.045251 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:50.045827 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:50.045834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:50.045913 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:50.046421 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:50.046435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:50.047431 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:50.047434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:50.048610 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:50.048625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:50.049281 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:50.049285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:50.049463 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:50.050030 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:50.050032 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:50.050066 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:50.050127 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:50.050417 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:50.050424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:50.050447 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:22:50.050584 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 23:22:50.051031 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":349,"threads_started":1}
I20260620 23:22:50.053166 10159 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37631
I20260620 23:22:50.053421 10159 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:50.054168 10159 heartbeater.cc:511] Master 127.0.97.62:37631 requested a full tablet report, sending...
I20260620 23:22:50.055433 9712 ts_manager.cc:194] Registered new tserver with Master: d7812929d04944aba5cf01f92f575dfc (127.0.97.3:36177)
I20260620 23:22:50.056187 9712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:50429
I20260620 23:22:50.057406 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:22:50.063557 9900 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:50.062173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.0.97.62:37631)
0620 23:22:50.062309 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:50.062312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:50.062844 (+ 532us) server_negotiation.cc:407] Connection header received
0620 23:22:50.062964 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:50.062968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:50.063026 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:50.063099 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:50.063413 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47548: BlockingRecv error: recv got EOF from 127.0.0.1:47548 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:22:50.064191 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 9771
I20260620 23:22:50.071038 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:50.065063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.0.97.1:37833)
0620 23:22:50.065566 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:50.065577 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:50.065599 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:22:50.065644 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:50.065649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:50.065844 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:50.065952 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:50.066234 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:50.067226 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:50.068313 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:50.068793 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:50.068896 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:50.069787 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:50.069794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:50.069804 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:50.070293 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:50.070330 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:50.070346 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:50.070445 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:50.070553 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 23:22:50.070683 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":282,"threads_started":1}
I20260620 23:22:50.072193 9893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:50.151669 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 9771
I20260620 23:22:50.167625 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 9902
I20260620 23:22:50.178268 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:50.168428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.0.97.2:39535)
0620 23:22:50.168918 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:50.168925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:50.168949 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:22:50.169002 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:50.169008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:50.169223 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:50.169352 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:50.169673 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:50.170633 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:50.171818 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:50.172347 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:50.172433 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:50.176956 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:50.176963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:50.176976 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:50.177534 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:50.177574 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:50.177591 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:50.177672 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:50.177801 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 23:22:50.177939 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":264,"threads_started":1}
I20260620 23:22:50.179499 10024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:50.256778 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 9902
I20260620 23:22:50.271970 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 10032
I20260620 23:22:50.280041 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:50.272918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.0.97.3:36177)
0620 23:22:50.273525 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:50.273531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:50.273559 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:22:50.273614 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:50.273621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:50.273856 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:50.274026 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:50.274457 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:50.275583 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:50.276771 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:50.277307 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:50.277445 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:50.278505 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:50.278513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:50.278526 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:50.279262 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:50.279305 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:50.279324 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:50.279439 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:50.279586 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 23:22:50.279730 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":329,"threads_started":1}
I20260620 23:22:50.281263 10154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:50.359127 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 10032
I20260620 23:22:50.375133 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 9681
I20260620 23:22:50.376183 9742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:50.472186 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 9681
2026-06-20T23:22:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2296 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T23:22:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:50Z Disabled control of system clock
I20260620 23:22:50.539108 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46059
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:22:50.844256 10178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:50.844596 10178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:50.844726 10178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:50.851624 10178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:50.851749 10178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:50.851837 10178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:50.851934 10178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:50.852013 10178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:22:50.860574 10178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:50.861763 10178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:50.863435 10178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:50.875285 10183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:50.875324 10184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:50.875345 10186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:50.876932 10178 server_base.cc:1034] running on GCE node
I20260620 23:22:50.877565 10178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:50.878858 10178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:50.880044 10178 hybrid_clock.cc:648] HybridClock initialized: now 1781997770880008 us; error 33 us; skew 500 ppm
I20260620 23:22:50.880359 10178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:50.882447 10178 webserver.cc:460] Webserver started at http://127.0.97.62:42023/ using document root <none> and password file <none>
I20260620 23:22:50.883117 10178 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:50.883225 10178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:50.883533 10178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:50.886119 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "f515310901f34fb09082e71b5a1bab21"
format_stamp: "Formatted at 2026-06-20 23:22:50 on dist-test-slave-5kzz"
server_key: "c203015e4ded1b81b60fb7cf27854ed5"
server_key_iv: "5d46751a124428ca73a83ef34e95e791"
server_key_version: "encryptionkey@0"
I20260620 23:22:50.886945 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "f515310901f34fb09082e71b5a1bab21"
format_stamp: "Formatted at 2026-06-20 23:22:50 on dist-test-slave-5kzz"
server_key: "c203015e4ded1b81b60fb7cf27854ed5"
server_key_iv: "5d46751a124428ca73a83ef34e95e791"
server_key_version: "encryptionkey@0"
I20260620 23:22:50.891744 10178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:22:50.894858 10192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:50.896055 10178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 23:22:50.896225 10178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "f515310901f34fb09082e71b5a1bab21"
format_stamp: "Formatted at 2026-06-20 23:22:50 on dist-test-slave-5kzz"
server_key: "c203015e4ded1b81b60fb7cf27854ed5"
server_key_iv: "5d46751a124428ca73a83ef34e95e791"
server_key_version: "encryptionkey@0"
I20260620 23:22:50.896430 10178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:50.901597 10178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:50.902304 10178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:50.902594 10178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:50.919376 10178 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46059
I20260620 23:22:50.919397 10243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46059 every 8 connection(s)
I20260620 23:22:50.920572 10178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:50.923887 10244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:50.926072 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 10178
I20260620 23:22:50.926270 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:22:50.927022 388 external_mini_cluster.cc:1428] Setting key e8292b7467c731ab9c259de50daf64ff
I20260620 23:22:50.935034 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:50.929018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.0.97.62:46059)
0620 23:22:50.930005 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:50.930014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:50.930051 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:22:50.931668 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:50.931722 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:50.932081 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:50.932449 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:50.933455 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:50.933460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:50.933472 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:50.933484 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:50.933545 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:50.933567 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:50.933782 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:50.933916 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 23:22:50.934161 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":388,"threads_started":1}
I20260620 23:22:50.935518 10244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21: Bootstrap starting.
I20260620 23:22:50.938814 10244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:50.940086 10244 log.cc:824] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:50.942464 10244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21: No bootstrap required, opened a new log
I20260620 23:22:50.949436 10244 raft_consensus.cc:348] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f515310901f34fb09082e71b5a1bab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46059 } }
I20260620 23:22:50.949765 10244 raft_consensus.cc:374] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:50.949869 10244 raft_consensus.cc:729] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f515310901f34fb09082e71b5a1bab21, State: Initialized, Role: FOLLOWER
I20260620 23:22:50.950374 10244 consensus_queue.cc:260] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f515310901f34fb09082e71b5a1bab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46059 } }
I20260620 23:22:50.950603 10244 raft_consensus.cc:388] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:50.950759 10244 raft_consensus.cc:482] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:50.950940 10244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:50.952952 10244 raft_consensus.cc:504] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f515310901f34fb09082e71b5a1bab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46059 } }
I20260620 23:22:50.953426 10244 leader_election.cc:302] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f515310901f34fb09082e71b5a1bab21; no voters:
I20260620 23:22:50.954397 10244 leader_election.cc:288] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:22:50.954493 10249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:50.955130 10249 raft_consensus.cc:686] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [term 1 LEADER]: Becoming Leader. State: Replica: f515310901f34fb09082e71b5a1bab21, State: Running, Role: LEADER
I20260620 23:22:50.955730 10249 consensus_queue.cc:237] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f515310901f34fb09082e71b5a1bab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46059 } }
I20260620 23:22:50.956578 10244 sys_catalog.cc:564] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:50.958670 10251 sys_catalog.cc:455] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f515310901f34fb09082e71b5a1bab21. Latest consensus state: current_term: 1 leader_uuid: "f515310901f34fb09082e71b5a1bab21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f515310901f34fb09082e71b5a1bab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46059 } } }
I20260620 23:22:50.958642 10250 sys_catalog.cc:455] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f515310901f34fb09082e71b5a1bab21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f515310901f34fb09082e71b5a1bab21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46059 } } }
I20260620 23:22:50.959060 10251 sys_catalog.cc:458] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:50.959121 10250 sys_catalog.cc:458] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21 [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:50.960568 10255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:50.965369 10255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:50.973846 10255 catalog_manager.cc:1347] Generated new cluster ID: b318650fb7414ce590ef8b25cd9aefd1
I20260620 23:22:50.973953 10255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:51.004570 10255 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:22:51.004959 10255 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:51.019620 10255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f515310901f34fb09082e71b5a1bab21: Generated new TSK 0
I20260620 23:22:51.020409 10255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:22:51.032130 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46059
--builtin_ntp_servers=127.0.97.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:22:51.343883 10268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:51.344128 10268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:51.344295 10268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:51.351212 10268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:51.351336 10268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:51.351505 10268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:22:51.360249 10268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:51.361449 10268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:51.363293 10268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:51.375649 10273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:51.375682 10276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:51.375576 10274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:51.376406 10268 server_base.cc:1034] running on GCE node
I20260620 23:22:51.376942 10268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:51.377892 10268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:51.379104 10268 hybrid_clock.cc:648] HybridClock initialized: now 1781997771379058 us; error 58 us; skew 500 ppm
I20260620 23:22:51.379410 10268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:51.381551 10268 webserver.cc:460] Webserver started at http://127.0.97.1:39327/ using document root <none> and password file <none>
I20260620 23:22:51.382220 10268 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:51.382354 10268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:51.382651 10268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:51.385365 10268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "7740964cc4fa4a1aa7f5fc470f152944"
format_stamp: "Formatted at 2026-06-20 23:22:51 on dist-test-slave-5kzz"
server_key: "426c0585159e5d00b8ad4ea2c875bb2f"
server_key_iv: "af9b7d83e45fe9a7aeba0c1a1a339fef"
server_key_version: "encryptionkey@0"
I20260620 23:22:51.386114 10268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7740964cc4fa4a1aa7f5fc470f152944"
format_stamp: "Formatted at 2026-06-20 23:22:51 on dist-test-slave-5kzz"
server_key: "426c0585159e5d00b8ad4ea2c875bb2f"
server_key_iv: "af9b7d83e45fe9a7aeba0c1a1a339fef"
server_key_version: "encryptionkey@0"
I20260620 23:22:51.391187 10268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:22:51.394300 10282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:51.395519 10268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:22:51.395678 10268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "7740964cc4fa4a1aa7f5fc470f152944"
format_stamp: "Formatted at 2026-06-20 23:22:51 on dist-test-slave-5kzz"
server_key: "426c0585159e5d00b8ad4ea2c875bb2f"
server_key_iv: "af9b7d83e45fe9a7aeba0c1a1a339fef"
server_key_version: "encryptionkey@0"
I20260620 23:22:51.395893 10268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:51.401261 10268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:51.401978 10268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:51.402251 10268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:51.403218 10268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:51.404480 10268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:51.404604 10268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:51.404721 10268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:51.404809 10268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:51.436008 10268 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39245
I20260620 23:22:51.436057 10394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39245 every 8 connection(s)
I20260620 23:22:51.437319 10268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:22:51.443992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 10268
I20260620 23:22:51.444205 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:22:51.444787 388 external_mini_cluster.cc:1428] Setting key 68462faf3fb4772a92876488e25f9105
I20260620 23:22:51.445384 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:51.439811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59041 (local address 127.0.97.62:46059)
0620 23:22:51.440241 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:51.440245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:51.441300 (+ 1055us) server_negotiation.cc:407] Connection header received
0620 23:22:51.442999 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:51.443005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:51.443086 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:51.443215 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:51.444595 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:51.444601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:51.444605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:51.444622 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:51.444668 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:51.444680 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:51.444805 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:51.444994 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 23:22:51.445070 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":240,"threads_started":1}
I20260620 23:22:51.446254 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:51.440284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46059 (local address 127.0.97.1:59041)
0620 23:22:51.441146 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:51.441185 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:51.442720 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:51.443472 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:51.443480 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:51.443568 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:51.444271 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:51.444274 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:51.444346 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:51.444439 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:51.444777 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:51.444787 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:51.444794 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:22:51.445089 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 23:22:51.445364 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":272,"threads_started":1}
I20260620 23:22:51.447539 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46059
--builtin_ntp_servers=127.0.97.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 23:22:51.447666 10395 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46059
I20260620 23:22:51.447947 10395 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:51.448585 10395 heartbeater.cc:511] Master 127.0.97.62:46059 requested a full tablet report, sending...
I20260620 23:22:51.450577 10209 ts_manager.cc:194] Registered new tserver with Master: 7740964cc4fa4a1aa7f5fc470f152944 (127.0.97.1:39245)
W20260620 23:22:51.759681 10399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:51.759930 10399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:51.760025 10399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:51.766862 10399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:51.766944 10399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:51.767042 10399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:22:51.775238 10399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:51.776315 10399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:51.777911 10399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:51.789966 10407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:51.790064 10404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:51.789999 10405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:51.790913 10399 server_base.cc:1034] running on GCE node
I20260620 23:22:51.791518 10399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:51.792356 10399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:51.793589 10399 hybrid_clock.cc:648] HybridClock initialized: now 1781997771793550 us; error 53 us; skew 500 ppm
I20260620 23:22:51.793900 10399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:51.795961 10399 webserver.cc:460] Webserver started at http://127.0.97.2:44849/ using document root <none> and password file <none>
I20260620 23:22:51.796577 10399 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:51.796700 10399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:51.796993 10399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:51.801445 10399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "40610ae6c07043489d16a9d42ae35f48"
format_stamp: "Formatted at 2026-06-20 23:22:51 on dist-test-slave-5kzz"
server_key: "8047672aec65ed16323097dee59d0e59"
server_key_iv: "9a21991f0ea52d660a6bd19987c47dfc"
server_key_version: "encryptionkey@0"
I20260620 23:22:51.802170 10399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "40610ae6c07043489d16a9d42ae35f48"
format_stamp: "Formatted at 2026-06-20 23:22:51 on dist-test-slave-5kzz"
server_key: "8047672aec65ed16323097dee59d0e59"
server_key_iv: "9a21991f0ea52d660a6bd19987c47dfc"
server_key_version: "encryptionkey@0"
I20260620 23:22:51.806828 10399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 23:22:51.809913 10413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:51.811203 10399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:22:51.811360 10399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "40610ae6c07043489d16a9d42ae35f48"
format_stamp: "Formatted at 2026-06-20 23:22:51 on dist-test-slave-5kzz"
server_key: "8047672aec65ed16323097dee59d0e59"
server_key_iv: "9a21991f0ea52d660a6bd19987c47dfc"
server_key_version: "encryptionkey@0"
I20260620 23:22:51.811571 10399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:51.816718 10399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:51.817404 10399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:51.817652 10399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:51.818464 10399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:51.819703 10399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:51.819818 10399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:51.819947 10399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:51.820032 10399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:51.850947 10399 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40021
I20260620 23:22:51.850992 10525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40021 every 8 connection(s)
I20260620 23:22:51.852210 10399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:22:51.858314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 10399
I20260620 23:22:51.858490 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:22:51.858976 388 external_mini_cluster.cc:1428] Setting key aa6d4d00c64fc73c181abdf4cfb72473
I20260620 23:22:51.859846 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:51.854550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51509 (local address 127.0.97.62:46059)
0620 23:22:51.854767 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:51.854772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:51.856069 (+ 1297us) server_negotiation.cc:407] Connection header received
0620 23:22:51.857676 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:51.857682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:51.857749 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:51.857859 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:51.859157 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:51.859162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:51.859167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:51.859187 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:51.859224 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:51.859239 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:51.859357 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:51.859525 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 23:22:51.859583 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:22:51.860803 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:51.855096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46059 (local address 127.0.97.2:51509)
0620 23:22:51.855928 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:51.855965 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:51.857414 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:51.858062 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:51.858070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:51.858152 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:51.858856 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:51.858858 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:51.858890 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:51.858965 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:51.859346 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:51.859352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:51.859357 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:22:51.859576 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 23:22:51.859816 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":259,"threads_started":1}
I20260620 23:22:51.861752 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46059
--builtin_ntp_servers=127.0.97.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 23:22:51.862139 10526 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46059
I20260620 23:22:51.862438 10526 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:51.863157 10526 heartbeater.cc:511] Master 127.0.97.62:46059 requested a full tablet report, sending...
I20260620 23:22:51.864622 10209 ts_manager.cc:194] Registered new tserver with Master: 40610ae6c07043489d16a9d42ae35f48 (127.0.97.2:40021)
W20260620 23:22:52.177736 10529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:52.178025 10529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:52.178113 10529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:52.185266 10529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:52.185344 10529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:52.185444 10529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:22:52.194887 10529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:52.196026 10529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:52.197638 10529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:52.210556 10537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:52.210512 10535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:52.210806 10534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:52.211529 10529 server_base.cc:1034] running on GCE node
I20260620 23:22:52.212149 10529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:52.213032 10529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:52.214244 10529 hybrid_clock.cc:648] HybridClock initialized: now 1781997772214206 us; error 52 us; skew 500 ppm
I20260620 23:22:52.214541 10529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:22:52.216748 10529 webserver.cc:460] Webserver started at http://127.0.97.3:39645/ using document root <none> and password file <none>
I20260620 23:22:52.217386 10529 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:52.217512 10529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:52.217824 10529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:52.220528 10529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "e81f83e0a00d4847850e6783583fa59f"
format_stamp: "Formatted at 2026-06-20 23:22:52 on dist-test-slave-5kzz"
server_key: "94c241597ec2a802e914b23a336be32a"
server_key_iv: "3591b2a3554e8d63eaac93d894eb7450"
server_key_version: "encryptionkey@0"
I20260620 23:22:52.221280 10529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e81f83e0a00d4847850e6783583fa59f"
format_stamp: "Formatted at 2026-06-20 23:22:52 on dist-test-slave-5kzz"
server_key: "94c241597ec2a802e914b23a336be32a"
server_key_iv: "3591b2a3554e8d63eaac93d894eb7450"
server_key_version: "encryptionkey@0"
I20260620 23:22:52.225914 10529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 23:22:52.228940 10543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:52.230082 10529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:22:52.230244 10529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "e81f83e0a00d4847850e6783583fa59f"
format_stamp: "Formatted at 2026-06-20 23:22:52 on dist-test-slave-5kzz"
server_key: "94c241597ec2a802e914b23a336be32a"
server_key_iv: "3591b2a3554e8d63eaac93d894eb7450"
server_key_version: "encryptionkey@0"
I20260620 23:22:52.230469 10529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:52.235746 10529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:52.236541 10529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:52.236824 10529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:52.237668 10529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:52.238950 10529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:52.239064 10529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:52.239189 10529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:52.239274 10529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:52.269776 10529 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46869
I20260620 23:22:52.269820 10655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46869 every 8 connection(s)
I20260620 23:22:52.271157 10529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:22:52.273136 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 10529
I20260620 23:22:52.273339 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:22:52.273821 388 external_mini_cluster.cc:1428] Setting key bee86b7354e88228c33e98101941c900
I20260620 23:22:52.279500 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:52.273825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38505 (local address 127.0.97.62:46059)
0620 23:22:52.274063 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:52.274069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:52.275840 (+ 1771us) server_negotiation.cc:407] Connection header received
0620 23:22:52.277657 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:52.277660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:52.277705 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:52.277780 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:52.279001 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:52.279004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:52.279007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:52.279018 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:52.279043 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:52.279052 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:52.279146 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:52.279276 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 23:22:52.279318 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 23:22:52.280370 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:52.274460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46059 (local address 127.0.97.3:38505)
0620 23:22:52.275655 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:52.275697 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:52.277441 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:52.277972 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:52.277979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:52.278061 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:52.278747 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:22:52.278751 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:52.278795 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:22:52.278875 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:52.279122 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:52.279127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:52.279131 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:22:52.279319 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 23:22:52.279491 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":395,"threads_started":1}
I20260620 23:22:52.281472 10656 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46059
I20260620 23:22:52.281746 10656 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:52.282256 10656 heartbeater.cc:511] Master 127.0.97.62:46059 requested a full tablet report, sending...
I20260620 23:22:52.283643 10209 ts_manager.cc:194] Registered new tserver with Master: e81f83e0a00d4847850e6783583fa59f (127.0.97.3:46869)
I20260620 23:22:52.288594 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:22:52.294281 10397 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:22:52.292847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.0.97.62:46059)
0620 23:22:52.293042 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:52.293050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:52.293497 (+ 447us) server_negotiation.cc:407] Connection header received
0620 23:22:52.293612 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:52.293615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:52.293659 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:52.293734 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:52.294060 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53094: BlockingRecv error: recv got EOF from 127.0.0.1:53094 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:22:52.294656 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 10268
I20260620 23:22:52.297964 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:52.295674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.0.97.1:39245)
0620 23:22:52.296157 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:52.296163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:52.296187 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:22:52.296237 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:52.296242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:52.296428 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:52.296541 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:52.297359 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:52.297365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:52.297376 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:52.297388 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:52.297429 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:52.297445 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:52.297553 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:52.297599 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 23:22:52.297703 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":233,"threads_started":1}
I20260620 23:22:52.299151 10390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:52.378943 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 10268
I20260620 23:22:52.395133 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 10399
I20260620 23:22:52.398442 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:52.396036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.0.97.2:40021)
0620 23:22:52.396461 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:52.396468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:52.396492 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:22:52.396550 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:52.396557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:52.396802 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:52.396922 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:52.397788 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:52.397794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:52.397805 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:52.397821 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:52.397858 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:52.397872 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:52.397984 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:52.398058 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 23:22:52.398168 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":192,"threads_started":1}
I20260620 23:22:52.399695 10521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:52.478101 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 10399
I20260620 23:22:52.493782 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 10529
I20260620 23:22:52.497495 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:52.494777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.0.97.3:46869)
0620 23:22:52.495411 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:52.495424 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:52.495453 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:22:52.495511 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:52.495517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:52.495721 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:52.495847 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:52.496837 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:52.496842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:52.496853 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:22:52.496867 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:52.496902 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:22:52.496915 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:52.497043 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:52.497089 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 23:22:52.497197 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":341,"threads_started":1}
I20260620 23:22:52.498647 10651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:52.579453 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 10529
I20260620 23:22:52.600631 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 10178
I20260620 23:22:52.602182 10239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:52.706075 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 10178
2026-06-20T23:22:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2233 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:22:52 dist-test-slave-5kzz krb5kdc[10674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:22:52 dist-test-slave-5kzz krb5kdc[10674](info): set up 2 sockets
Jun 20 23:22:52 dist-test-slave-5kzz krb5kdc[10674](info): commencing operation
krb5kdc: starting...
W20260620 23:22:53.779394 388 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:22:53 dist-test-slave-5kzz krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:22:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:22:53.982999 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44245
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:54.289575 10687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:54.289868 10687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:54.290012 10687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:54.297078 10687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:22:54.297216 10687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:54.297309 10687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:54.297400 10687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:22:54.297477 10687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:22:54.307319 10687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:54.308530 10687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:54.309721 10687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:54.320559 10693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:54.320645 10692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:54.320621 10695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:54.321524 10687 server_base.cc:1034] running on GCE node
I20260620 23:22:54.322240 10687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:54.323529 10687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:54.324716 10687 hybrid_clock.cc:648] HybridClock initialized: now 1781997774324657 us; error 52 us; skew 500 ppm
I20260620 23:22:54.325038 10687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:54 dist-test-slave-5kzz krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:54.328011 10687 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:22:54.329382 10687 webserver.cc:460] Webserver started at http://127.0.97.62:38495/ using document root <none> and password file <none>
I20260620 23:22:54.330003 10687 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:54.330139 10687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:54.330420 10687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:54.333225 10687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "cbbf3f4b0984435fbf5507434b9ffccf"
format_stamp: "Formatted at 2026-06-20 23:22:54 on dist-test-slave-5kzz"
server_key: "2d81fd954d682bffba190275d6263460"
server_key_iv: "9febee1dac660acc7efda146d510f091"
server_key_version: "encryptionkey@0"
I20260620 23:22:54.333923 10687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "cbbf3f4b0984435fbf5507434b9ffccf"
format_stamp: "Formatted at 2026-06-20 23:22:54 on dist-test-slave-5kzz"
server_key: "2d81fd954d682bffba190275d6263460"
server_key_iv: "9febee1dac660acc7efda146d510f091"
server_key_version: "encryptionkey@0"
I20260620 23:22:54.338586 10687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 23:22:54.341697 10702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:54.342998 10687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:22:54.343173 10687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "cbbf3f4b0984435fbf5507434b9ffccf"
format_stamp: "Formatted at 2026-06-20 23:22:54 on dist-test-slave-5kzz"
server_key: "2d81fd954d682bffba190275d6263460"
server_key_iv: "9febee1dac660acc7efda146d510f091"
server_key_version: "encryptionkey@0"
I20260620 23:22:54.343346 10687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:54.355662 10687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:54.359464 10687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:54.359732 10687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:54.377596 10687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44245
I20260620 23:22:54.377624 10753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44245 every 8 connection(s)
I20260620 23:22:54.378943 10687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:22:54.382496 10754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:54.385165 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 10687
I20260620 23:22:54.385392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:22:54.386145 388 external_mini_cluster.cc:1428] Setting key 07abd7bf674201d59033285ffc0c1e4a
I20260620 23:22:54.393910 10754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf: Bootstrap starting.
I20260620 23:22:54.397085 10754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:54.398092 10754 log.cc:824] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:22:54 dist-test-slave-5kzz krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:22:54.401311 10754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf: No bootstrap required, opened a new log
I20260620 23:22:54.408198 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:54.388278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46068 (local address 127.0.97.62:44245)
0620 23:22:54.388970 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:54.388981 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:54.389019 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:22:54.390731 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:54.390759 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:54.391139 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:54.391510 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:54.392998 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:54.394350 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:54.395650 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:54.396401 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:54.396593 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:54.401256 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:54.401264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:54.401284 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:54.401320 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:54.404403 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:54.405059 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:54.405065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:54.405071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:54.405218 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:54.405629 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:54.405634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:54.405637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:54.406131 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:54.406446 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:54.406860 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 23:22:54.407253 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":310,"threads_started":1}
I20260620 23:22:54.409600 10754 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbf3f4b0984435fbf5507434b9ffccf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44245 } }
I20260620 23:22:54.409976 10754 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:54.410096 10754 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbbf3f4b0984435fbf5507434b9ffccf, State: Initialized, Role: FOLLOWER
I20260620 23:22:54.410633 10754 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbf3f4b0984435fbf5507434b9ffccf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44245 } }
I20260620 23:22:54.410940 10754 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:22:54.411095 10754 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:22:54.411279 10754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:54.413528 10754 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbf3f4b0984435fbf5507434b9ffccf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44245 } }
I20260620 23:22:54.413995 10754 leader_election.cc:302] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbbf3f4b0984435fbf5507434b9ffccf; no voters:
I20260620 23:22:54.414553 10754 leader_election.cc:288] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:22:54.414705 10759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:54.415388 10759 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [term 1 LEADER]: Becoming Leader. State: Replica: cbbf3f4b0984435fbf5507434b9ffccf, State: Running, Role: LEADER
I20260620 23:22:54.415928 10759 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbf3f4b0984435fbf5507434b9ffccf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44245 } }
I20260620 23:22:54.416461 10754 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:22:54.418567 10760 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbbf3f4b0984435fbf5507434b9ffccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbf3f4b0984435fbf5507434b9ffccf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44245 } } }
I20260620 23:22:54.418726 10761 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbbf3f4b0984435fbf5507434b9ffccf. Latest consensus state: current_term: 1 leader_uuid: "cbbf3f4b0984435fbf5507434b9ffccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbf3f4b0984435fbf5507434b9ffccf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44245 } } }
I20260620 23:22:54.419054 10761 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:54.421348 10760 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf [sys.catalog]: This master's current role is: LEADER
I20260620 23:22:54.423894 10765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:22:54.428756 10765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:22:54.437620 10765 catalog_manager.cc:1347] Generated new cluster ID: 53c5c5140b474d269a20e4391a636580
I20260620 23:22:54.437727 10765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:22:54.460026 10765 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:22:54.461789 10765 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:22:54.472078 10765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf: Generated new TSK 0
I20260620 23:22:54.473117 10765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:22:54.527921 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44245
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:54.845501 10780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:54.845846 10780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:54.846016 10780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:54.853152 10780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:54.853294 10780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:54.853456 10780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:22:54.862609 10780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:54.863837 10780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:54.865011 10780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:54.877859 10786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:54.878005 10788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:54.878017 10785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:54.878790 10780 server_base.cc:1034] running on GCE node
I20260620 23:22:54.879380 10780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:54.880167 10780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:54.881347 10780 hybrid_clock.cc:648] HybridClock initialized: now 1781997774881294 us; error 51 us; skew 500 ppm
I20260620 23:22:54.881676 10780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:54 dist-test-slave-5kzz krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:54.884583 10780 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:22:54.885838 10780 webserver.cc:460] Webserver started at http://127.0.97.1:33789/ using document root <none> and password file <none>
I20260620 23:22:54.886479 10780 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:54.886615 10780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:54.886924 10780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:54.889572 10780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "8cf25f34585a4224aae4dd6355bdcc6b"
format_stamp: "Formatted at 2026-06-20 23:22:54 on dist-test-slave-5kzz"
server_key: "5a731d1787152121206d1de5787124a7"
server_key_iv: "de31fce2396165e68a6de9a1ec6c0292"
server_key_version: "encryptionkey@0"
I20260620 23:22:54.890281 10780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8cf25f34585a4224aae4dd6355bdcc6b"
format_stamp: "Formatted at 2026-06-20 23:22:54 on dist-test-slave-5kzz"
server_key: "5a731d1787152121206d1de5787124a7"
server_key_iv: "de31fce2396165e68a6de9a1ec6c0292"
server_key_version: "encryptionkey@0"
I20260620 23:22:54.895197 10780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:22:54.898463 10795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:54.899855 10780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:22:54.900012 10780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "8cf25f34585a4224aae4dd6355bdcc6b"
format_stamp: "Formatted at 2026-06-20 23:22:54 on dist-test-slave-5kzz"
server_key: "5a731d1787152121206d1de5787124a7"
server_key_iv: "de31fce2396165e68a6de9a1ec6c0292"
server_key_version: "encryptionkey@0"
I20260620 23:22:54.900203 10780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:54.922870 10780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:54.926842 10780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:54.927106 10780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:54.928097 10780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:54.929446 10780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:54.929548 10780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:54.929685 10780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:54.929755 10780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:54.961824 10780 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41995
I20260620 23:22:54.961874 10907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41995 every 8 connection(s)
I20260620 23:22:54.963146 10780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:22:54.970731 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 10780
I20260620 23:22:54.970990 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:22:54.971681 388 external_mini_cluster.cc:1428] Setting key 7059373dad3f0b0b0a4737cf525b0e8d
Jun 20 23:22:54 dist-test-slave-5kzz krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:22:54.982605 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:54.965625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32811 (local address 127.0.97.62:44245)
0620 23:22:54.966038 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:54.966044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:54.967074 (+ 1030us) server_negotiation.cc:407] Connection header received
0620 23:22:54.968786 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:54.968791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:54.968854 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:54.968958 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:54.970349 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:54.971252 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:54.972989 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:54.973409 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:54.973537 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:54.976953 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:54.976958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:54.976961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:54.976990 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:54.979769 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:54.980455 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:54.980461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:54.980465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:54.980574 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:54.980994 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:54.981001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:54.981004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:54.981272 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:54.981423 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:54.982050 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 23:22:54.982253 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":200,"threads_started":1}
I20260620 23:22:54.983605 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:54.966203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44245 (local address 127.0.97.1:32811)
0620 23:22:54.966918 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:54.966961 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:54.968543 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:54.969150 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:54.969157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:54.969654 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:54.970184 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:54.970199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:54.971391 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:54.971398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:54.972853 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:54.972864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:54.973613 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:54.973620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:54.973880 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:54.974584 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:54.974612 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:54.976738 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:54.979920 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:54.979929 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:54.979946 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:54.980338 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:54.980679 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:54.980684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:54.980688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:54.980858 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:54.981399 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:54.981407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:54.981721 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 23:22:54.981980 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 23:22:54.982494 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":255,"threads_started":1}
I20260620 23:22:54.985088 10908 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44245
I20260620 23:22:54.985378 10908 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:54.986052 10908 heartbeater.cc:511] Master 127.0.97.62:44245 requested a full tablet report, sending...
I20260620 23:22:54.988265 10719 ts_manager.cc:194] Registered new tserver with Master: 8cf25f34585a4224aae4dd6355bdcc6b (127.0.97.1:41995)
I20260620 23:22:54.990409 10719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32811
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:22:55.019634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44245
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:55.329806 10914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:55.330045 10914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:55.330210 10914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:55.337255 10914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:55.337360 10914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:55.337532 10914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:22:55.347020 10914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:55.348304 10914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:55.349632 10914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:55.362195 10919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:55.362165 10922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:55.362231 10920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:55.363309 10914 server_base.cc:1034] running on GCE node
I20260620 23:22:55.363906 10914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:55.364812 10914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:55.365998 10914 hybrid_clock.cc:648] HybridClock initialized: now 1781997775365962 us; error 34 us; skew 500 ppm
I20260620 23:22:55.366291 10914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:55 dist-test-slave-5kzz krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:55.373517 10914 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:22:55.374927 10914 webserver.cc:460] Webserver started at http://127.0.97.2:42921/ using document root <none> and password file <none>
I20260620 23:22:55.375567 10914 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:55.375706 10914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:55.376004 10914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:55.378757 10914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "bfcc372f4048444fb9454b78cb3c754b"
format_stamp: "Formatted at 2026-06-20 23:22:55 on dist-test-slave-5kzz"
server_key: "33442e8be3f2a86f1f72a2111f6ed43f"
server_key_iv: "7c367676634f3d758c54827da665896b"
server_key_version: "encryptionkey@0"
I20260620 23:22:55.379463 10914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bfcc372f4048444fb9454b78cb3c754b"
format_stamp: "Formatted at 2026-06-20 23:22:55 on dist-test-slave-5kzz"
server_key: "33442e8be3f2a86f1f72a2111f6ed43f"
server_key_iv: "7c367676634f3d758c54827da665896b"
server_key_version: "encryptionkey@0"
I20260620 23:22:55.384280 10914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 23:22:55.387552 10929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:55.388797 10914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:22:55.388953 10914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "bfcc372f4048444fb9454b78cb3c754b"
format_stamp: "Formatted at 2026-06-20 23:22:55 on dist-test-slave-5kzz"
server_key: "33442e8be3f2a86f1f72a2111f6ed43f"
server_key_iv: "7c367676634f3d758c54827da665896b"
server_key_version: "encryptionkey@0"
I20260620 23:22:55.389137 10914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:55.400287 10914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:55.404454 10914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:55.404723 10914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:55.405550 10914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:55.406786 10914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:55.406883 10914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:55.407024 10914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:55.407092 10914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:55.442212 10914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43305
I20260620 23:22:55.442265 11041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43305 every 8 connection(s)
I20260620 23:22:55.443611 10914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:22:55.450017 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 10914
I20260620 23:22:55.450201 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:22:55.450637 388 external_mini_cluster.cc:1428] Setting key 196e04a1c9d882453558883b3544fe15
Jun 20 23:22:55 dist-test-slave-5kzz krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:22:55.465898 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:55.446094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46009 (local address 127.0.97.62:44245)
0620 23:22:55.446287 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:55.446293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:55.447621 (+ 1328us) server_negotiation.cc:407] Connection header received
0620 23:22:55.449306 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:55.449310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:55.449374 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:55.449463 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:55.451122 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:55.452170 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:55.454606 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:55.455048 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:55.455142 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:55.459341 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:55.459346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:55.459350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:55.459380 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:55.462845 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:55.463528 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:55.463536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:55.463542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:55.463698 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:55.464099 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:55.464106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:55.464112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:55.464463 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:55.464611 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:55.465229 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 23:22:55.465477 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:22:55.466512 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:55.446714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44245 (local address 127.0.97.2:46009)
0620 23:22:55.447455 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:55.447493 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:55.449052 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:55.449665 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:55.449673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:55.450231 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:55.450948 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:55.450967 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:55.452330 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:55.452336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:55.454456 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:55.454474 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:55.455276 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:55.455285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:55.455591 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:55.456281 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:55.456312 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:55.459119 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:55.462968 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:55.462975 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:55.462989 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:55.463399 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:55.463804 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:55.463808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:55.463810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:55.463978 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:55.464569 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:55.464575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:55.464873 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 23:22:55.465079 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 23:22:55.465444 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":253,"threads_started":1}
I20260620 23:22:55.467943 11042 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44245
I20260620 23:22:55.468256 11042 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:55.469135 11042 heartbeater.cc:511] Master 127.0.97.62:44245 requested a full tablet report, sending...
I20260620 23:22:55.470746 10719 ts_manager.cc:194] Registered new tserver with Master: bfcc372f4048444fb9454b78cb3c754b (127.0.97.2:43305)
I20260620 23:22:55.471659 10719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46009
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:22:55.501729 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44245
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:22:55.815711 11047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:22:55.816035 11047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:22:55.816201 11047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:22:55.823367 11047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:22:55.823508 11047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:22:55.823688 11047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:22:55.833076 11047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:22:55.834359 11047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:22:55.835675 11047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:22:55.848081 11055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:55.848198 11052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:22:55.848177 11053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:22:55.849217 11047 server_base.cc:1034] running on GCE node
I20260620 23:22:55.849787 11047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:22:55.850721 11047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:22:55.851964 11047 hybrid_clock.cc:648] HybridClock initialized: now 1781997775851923 us; error 49 us; skew 500 ppm
I20260620 23:22:55.852267 11047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:22:55 dist-test-slave-5kzz krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:22:55.855297 11047 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:22:55.856722 11047 webserver.cc:460] Webserver started at http://127.0.97.3:37941/ using document root <none> and password file <none>
I20260620 23:22:55.857424 11047 fs_manager.cc:359] Metadata directory not provided
I20260620 23:22:55.857566 11047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:22:55.857869 11047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:22:55.860790 11047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "f6a6b1160fa848019c342aeca364dc89"
format_stamp: "Formatted at 2026-06-20 23:22:55 on dist-test-slave-5kzz"
server_key: "d76ae84bd2943903291beb1ed3d64a73"
server_key_iv: "97162cd47d76725d45c754bbe09f5a7f"
server_key_version: "encryptionkey@0"
I20260620 23:22:55.861526 11047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f6a6b1160fa848019c342aeca364dc89"
format_stamp: "Formatted at 2026-06-20 23:22:55 on dist-test-slave-5kzz"
server_key: "d76ae84bd2943903291beb1ed3d64a73"
server_key_iv: "97162cd47d76725d45c754bbe09f5a7f"
server_key_version: "encryptionkey@0"
I20260620 23:22:55.866382 11047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260620 23:22:55.869649 11062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:55.871001 11047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:22:55.871167 11047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "f6a6b1160fa848019c342aeca364dc89"
format_stamp: "Formatted at 2026-06-20 23:22:55 on dist-test-slave-5kzz"
server_key: "d76ae84bd2943903291beb1ed3d64a73"
server_key_iv: "97162cd47d76725d45c754bbe09f5a7f"
server_key_version: "encryptionkey@0"
I20260620 23:22:55.871367 11047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:22:55.889621 11047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:22:55.893837 11047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:22:55.894137 11047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:22:55.895105 11047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:22:55.896458 11047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:22:55.896559 11047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:55.896703 11047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:22:55.896782 11047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:22:55.929693 11047 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37177
I20260620 23:22:55.929735 11174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37177 every 8 connection(s)
I20260620 23:22:55.931033 11047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:22:55.932590 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 11047
I20260620 23:22:55.932797 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:22:55.933466 388 external_mini_cluster.cc:1428] Setting key fd40c261f8be13290331c134f9fc6059
Jun 20 23:22:55 dist-test-slave-5kzz krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:22:55.950206 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:55.933931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40025 (local address 127.0.97.62:44245)
0620 23:22:55.934190 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:55.934199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:55.935541 (+ 1342us) server_negotiation.cc:407] Connection header received
0620 23:22:55.937348 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:55.937352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:55.937431 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:55.937522 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:55.938952 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:55.939699 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:55.941112 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:55.941491 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:55.941593 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:55.944793 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:55.944798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:55.944801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:55.944830 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:55.947412 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:55.948060 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:55.948065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:55.948068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:55.948201 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:55.948624 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:55.948629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:55.948632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:55.948882 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:55.949028 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:55.949667 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 23:22:55.949868 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 23:22:55.951136 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:55.934514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44245 (local address 127.0.97.3:40025)
0620 23:22:55.935392 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:55.935435 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:55.937106 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:55.937681 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:55.937689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:55.938178 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:22:55.938800 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:55.938819 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:55.939819 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:55.939822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:55.940953 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:55.940963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:55.941672 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:55.941677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:55.941862 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:55.942459 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:22:55.942484 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:22:55.944600 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:22:55.947569 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:55.947575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:55.947586 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:55.947951 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:55.948305 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:22:55.948309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:22:55.948313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:22:55.948504 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:22:55.949004 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:22:55.949011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:22:55.949319 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 23:22:55.949567 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 23:22:55.950040 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":377,"threads_started":1}
I20260620 23:22:55.952337 11175 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44245
I20260620 23:22:55.952622 11175 heartbeater.cc:464] Registering TS with master...
I20260620 23:22:55.953294 11175 heartbeater.cc:511] Master 127.0.97.62:44245 requested a full tablet report, sending...
I20260620 23:22:55.954598 10719 ts_manager.cc:194] Registered new tserver with Master: f6a6b1160fa848019c342aeca364dc89 (127.0.97.3:37177)
I20260620 23:22:55.955417 10719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40025
I20260620 23:22:55.960563 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:22:55.976200 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:55.965056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.0.97.62:44245)
0620 23:22:55.965250 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:55.965257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:55.965773 (+ 516us) server_negotiation.cc:407] Connection header received
0620 23:22:55.965890 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:55.965894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:55.965948 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:55.966025 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:55.967384 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:55.968040 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:55.969167 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:55.969576 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:55.969660 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:55.971789 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:55.971793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:55.971797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:55.971821 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:55.973985 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:55.974535 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:55.974539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:55.974541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:55.974728 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:55.975015 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:55.975018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:55.975021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:55.975310 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:55.975515 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:55.975768 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 23:22:55.975936 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:22:55.979939 10719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:22:55.986533 10719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:22:55.993326 10908 heartbeater.cc:503] Master 127.0.97.62:44245 was elected leader, sending a full tablet report...
I20260620 23:22:56.005750 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:55.998121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.0.97.2:43305)
0620 23:22:55.998663 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:55.998670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:55.998842 (+ 172us) server_negotiation.cc:407] Connection header received
0620 23:22:55.999223 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:55.999231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:55.999479 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:55.999677 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:56.000399 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.001504 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.003532 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.004648 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.004817 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.005024 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:56.005286 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 23:22:56.005441 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":306,"threads_started":1}
I20260620 23:22:56.005800 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:55.997938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43305 (local address 127.0.0.1:44192)
0620 23:22:55.998652 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:55.998717 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:55.998892 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:55.999679 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:55.999690 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:55.999722 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:56.000156 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:56.000170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.001632 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.001639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:56.003370 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:56.003388 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.004862 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.004869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:56.005066 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.005112 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 23:22:56.005248 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 23:22:56.005442 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":777,"threads_started":1}
I20260620 23:22:56.013118 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:55.999513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41995 (local address 127.0.0.1:59332)
0620 23:22:56.005998 (+ 6485us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:56.006015 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:56.006130 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:56.006750 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:56.006757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:56.006793 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:56.007021 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:56.007034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.008300 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.008309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:56.010534 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:56.010547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.012304 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.012309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:56.012482 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.012539 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 23:22:56.012649 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:22:56.012786 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6402,"thread_start_us":10091,"threads_started":1}
I20260620 23:22:56.013132 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:55.999817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.0.97.1:41995)
0620 23:22:56.000502 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:56.000513 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:56.006244 (+ 5731us) server_negotiation.cc:407] Connection header received
0620 23:22:56.006358 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:56.006365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:56.006550 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:56.006758 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:56.007175 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.008157 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.010725 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.012153 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.012294 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.012507 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:56.012692 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 23:22:56.012838 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":300,"threads_started":1}
I20260620 23:22:56.014091 10977 tablet_service.cc:1467] Processing CreateTablet for tablet cfcbdf4cfa18456395b1bc095ebeb522 (DEFAULT_TABLE table=test-table [id=49a3bd32cb7a4d2885630950d17a1e54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:56.015643 10977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfcbdf4cfa18456395b1bc095ebeb522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:56.016882 10843 tablet_service.cc:1467] Processing CreateTablet for tablet cfcbdf4cfa18456395b1bc095ebeb522 (DEFAULT_TABLE table=test-table [id=49a3bd32cb7a4d2885630950d17a1e54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:56.017675 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:56.010133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37177 (local address 127.0.0.1:32964)
0620 23:22:56.010308 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:56.010330 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:56.010443 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:56.011565 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:56.011570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:56.011596 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:56.011933 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:56.011944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.013916 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.013921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:56.015451 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:56.015462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.016839 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.016845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:56.017009 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.017048 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:22:56.017156 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:22:56.017360 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 23:22:56.017673 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:56.010228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.0.97.3:37177)
0620 23:22:56.011038 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:56.011043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:56.011068 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:22:56.011137 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:56.011142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:56.011369 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:56.011564 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:56.012116 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.013761 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.015598 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.016647 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.016795 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.017007 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:56.017212 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 23:22:56.017364 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":520,"threads_started":1}
I20260620 23:22:56.019080 10843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfcbdf4cfa18456395b1bc095ebeb522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:56.020632 11110 tablet_service.cc:1467] Processing CreateTablet for tablet cfcbdf4cfa18456395b1bc095ebeb522 (DEFAULT_TABLE table=test-table [id=49a3bd32cb7a4d2885630950d17a1e54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:22:56.021654 11110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfcbdf4cfa18456395b1bc095ebeb522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:22:56.030105 11192 tablet_bootstrap.cc:492] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b: Bootstrap starting.
I20260620 23:22:56.032797 11193 tablet_bootstrap.cc:492] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89: Bootstrap starting.
I20260620 23:22:56.034382 11192 tablet_bootstrap.cc:654] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:56.035324 11194 tablet_bootstrap.cc:492] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b: Bootstrap starting.
I20260620 23:22:56.035717 11192 log.cc:824] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:56.036572 11193 tablet_bootstrap.cc:654] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:56.037618 11193 log.cc:824] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:56.038492 11194 tablet_bootstrap.cc:654] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b: Neither blocks nor log segments found. Creating new log.
I20260620 23:22:56.038906 11192 tablet_bootstrap.cc:492] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b: No bootstrap required, opened a new log
I20260620 23:22:56.039168 11192 ts_tablet_manager.cc:1397] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 23:22:56.039700 11194 log.cc:824] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:22:56.040328 11193 tablet_bootstrap.cc:492] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89: No bootstrap required, opened a new log
I20260620 23:22:56.040616 11193 ts_tablet_manager.cc:1397] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 23:22:56.047166 11194 tablet_bootstrap.cc:492] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b: No bootstrap required, opened a new log
I20260620 23:22:56.047415 11194 ts_tablet_manager.cc:1397] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260620 23:22:56.048499 11193 raft_consensus.cc:348] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcc372f4048444fb9454b78cb3c754b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43305 } } peers { permanent_uuid: "8cf25f34585a4224aae4dd6355bdcc6b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41995 } } peers { permanent_uuid: "f6a6b1160fa848019c342aeca364dc89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } }
I20260620 23:22:56.048846 11193 raft_consensus.cc:374] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:56.048941 11193 raft_consensus.cc:729] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a6b1160fa848019c342aeca364dc89, State: Initialized, Role: FOLLOWER
I20260620 23:22:56.049531 11193 consensus_queue.cc:260] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcc372f4048444fb9454b78cb3c754b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43305 } } peers { permanent_uuid: "8cf25f34585a4224aae4dd6355bdcc6b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41995 } } peers { permanent_uuid: "f6a6b1160fa848019c342aeca364dc89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } }
I20260620 23:22:56.051433 11175 heartbeater.cc:503] Master 127.0.97.62:44245 was elected leader, sending a full tablet report...
I20260620 23:22:56.051936 11193 ts_tablet_manager.cc:1428] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89: Time spent starting tablet: real 0.011s user 0.002s sys 0.009s
I20260620 23:22:56.051786 11192 raft_consensus.cc:348] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcc372f4048444fb9454b78cb3c754b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43305 } } peers { permanent_uuid: "8cf25f34585a4224aae4dd6355bdcc6b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41995 } } peers { permanent_uuid: "f6a6b1160fa848019c342aeca364dc89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } }
I20260620 23:22:56.052241 11192 raft_consensus.cc:374] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:56.052664 11192 raft_consensus.cc:729] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfcc372f4048444fb9454b78cb3c754b, State: Initialized, Role: FOLLOWER
I20260620 23:22:56.053364 11192 consensus_queue.cc:260] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcc372f4048444fb9454b78cb3c754b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43305 } } peers { permanent_uuid: "8cf25f34585a4224aae4dd6355bdcc6b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41995 } } peers { permanent_uuid: "f6a6b1160fa848019c342aeca364dc89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } }
I20260620 23:22:56.054483 11042 heartbeater.cc:503] Master 127.0.97.62:44245 was elected leader, sending a full tablet report...
I20260620 23:22:56.055105 11192 ts_tablet_manager.cc:1428] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260620 23:22:56.058745 11194 raft_consensus.cc:348] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcc372f4048444fb9454b78cb3c754b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43305 } } peers { permanent_uuid: "8cf25f34585a4224aae4dd6355bdcc6b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41995 } } peers { permanent_uuid: "f6a6b1160fa848019c342aeca364dc89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } }
I20260620 23:22:56.059108 11194 raft_consensus.cc:374] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:22:56.059212 11194 raft_consensus.cc:729] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf25f34585a4224aae4dd6355bdcc6b, State: Initialized, Role: FOLLOWER
I20260620 23:22:56.059796 11194 consensus_queue.cc:260] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcc372f4048444fb9454b78cb3c754b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43305 } } peers { permanent_uuid: "8cf25f34585a4224aae4dd6355bdcc6b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41995 } } peers { permanent_uuid: "f6a6b1160fa848019c342aeca364dc89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } }
I20260620 23:22:56.061308 11194 ts_tablet_manager.cc:1428] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
W20260620 23:22:56.182657 11176 tablet.cc:2369] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:22:56.195530 11043 tablet.cc:2369] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:22:56.215457 10909 tablet.cc:2369] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:22:56.291270 11199 raft_consensus.cc:482] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:22:56.291564 11199 raft_consensus.cc:504] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcc372f4048444fb9454b78cb3c754b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43305 } } peers { permanent_uuid: "8cf25f34585a4224aae4dd6355bdcc6b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41995 } } peers { permanent_uuid: "f6a6b1160fa848019c342aeca364dc89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } }
I20260620 23:22:56.293399 11199 leader_election.cc:288] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cf25f34585a4224aae4dd6355bdcc6b (127.0.97.1:41995), f6a6b1160fa848019c342aeca364dc89 (127.0.97.3:37177)
I20260620 23:22:56.300076 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:56.293820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36155 (local address 127.0.97.1:41995)
0620 23:22:56.294054 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:56.294060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:56.294637 (+ 577us) server_negotiation.cc:407] Connection header received
0620 23:22:56.294862 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:56.294868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:56.294942 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:56.295045 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:56.295825 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.296514 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.298241 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.299085 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.299234 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.299344 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:56.299639 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 23:22:56.299844 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:22:56.300122 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:56.293784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41995 (local address 127.0.97.2:36155)
0620 23:22:56.294532 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:56.294553 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:56.294708 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:56.295238 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:56.295248 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:56.295273 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:56.295703 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:56.295714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.296654 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.296660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:56.298067 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:56.298078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.299289 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.299295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:56.299465 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.299487 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:22:56.299606 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:22:56.299730 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":459,"threads_started":1}
I20260620 23:22:56.300542 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:56.294909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59215 (local address 127.0.97.3:37177)
0620 23:22:56.295160 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:56.295166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:56.295535 (+ 369us) server_negotiation.cc:407] Connection header received
0620 23:22:56.295691 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:56.295694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:56.295748 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:56.295858 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:22:56.296371 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.297059 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.298719 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.299460 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.299618 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.299896 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:56.300177 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:22:56.300296 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:22:56.300608 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:56.294705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37177 (local address 127.0.97.2:59215)
0620 23:22:56.295429 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 23:22:56.295447 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:22:56.295574 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:22:56.295876 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:22:56.295881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:22:56.295905 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:22:56.296203 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:56.296211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.297197 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.297204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:56.298548 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:22:56.298559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.299815 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.299822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:22:56.300010 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.300033 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:22:56.300143 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:22:56.300282 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":393,"threads_started":1}
I20260620 23:22:56.301270 10863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcbdf4cfa18456395b1bc095ebeb522" candidate_uuid: "bfcc372f4048444fb9454b78cb3c754b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf25f34585a4224aae4dd6355bdcc6b" is_pre_election: true
I20260620 23:22:56.301788 10863 raft_consensus.cc:2454] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfcc372f4048444fb9454b78cb3c754b in term 0.
I20260620 23:22:56.301764 11130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcbdf4cfa18456395b1bc095ebeb522" candidate_uuid: "bfcc372f4048444fb9454b78cb3c754b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a6b1160fa848019c342aeca364dc89" is_pre_election: true
I20260620 23:22:56.302237 11130 raft_consensus.cc:2454] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfcc372f4048444fb9454b78cb3c754b in term 0.
I20260620 23:22:56.302398 10931 leader_election.cc:302] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cf25f34585a4224aae4dd6355bdcc6b, bfcc372f4048444fb9454b78cb3c754b; no voters:
I20260620 23:22:56.302830 11199 raft_consensus.cc:2781] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:22:56.302979 11199 raft_consensus.cc:482] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:22:56.303115 11199 raft_consensus.cc:3037] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:56.305282 11199 raft_consensus.cc:504] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcc372f4048444fb9454b78cb3c754b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43305 } } peers { permanent_uuid: "8cf25f34585a4224aae4dd6355bdcc6b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41995 } } peers { permanent_uuid: "f6a6b1160fa848019c342aeca364dc89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } }
I20260620 23:22:56.305783 11199 leader_election.cc:288] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [CANDIDATE]: Term 1 election: Requested vote from peers 8cf25f34585a4224aae4dd6355bdcc6b (127.0.97.1:41995), f6a6b1160fa848019c342aeca364dc89 (127.0.97.3:37177)
I20260620 23:22:56.306144 10863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcbdf4cfa18456395b1bc095ebeb522" candidate_uuid: "bfcc372f4048444fb9454b78cb3c754b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf25f34585a4224aae4dd6355bdcc6b"
I20260620 23:22:56.306191 11130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcbdf4cfa18456395b1bc095ebeb522" candidate_uuid: "bfcc372f4048444fb9454b78cb3c754b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a6b1160fa848019c342aeca364dc89"
I20260620 23:22:56.306350 10863 raft_consensus.cc:3037] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:56.306370 11130 raft_consensus.cc:3037] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:22:56.309461 11130 raft_consensus.cc:2454] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfcc372f4048444fb9454b78cb3c754b in term 1.
I20260620 23:22:56.309463 10863 raft_consensus.cc:2454] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfcc372f4048444fb9454b78cb3c754b in term 1.
I20260620 23:22:56.309827 10931 leader_election.cc:302] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cf25f34585a4224aae4dd6355bdcc6b, bfcc372f4048444fb9454b78cb3c754b; no voters:
I20260620 23:22:56.310118 11199 raft_consensus.cc:2781] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:22:56.310537 11199 raft_consensus.cc:686] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 1 LEADER]: Becoming Leader. State: Replica: bfcc372f4048444fb9454b78cb3c754b, State: Running, Role: LEADER
I20260620 23:22:56.311028 11199 consensus_queue.cc:237] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcc372f4048444fb9454b78cb3c754b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43305 } } peers { permanent_uuid: "8cf25f34585a4224aae4dd6355bdcc6b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41995 } } peers { permanent_uuid: "f6a6b1160fa848019c342aeca364dc89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } }
I20260620 23:22:56.315512 10718 catalog_manager.cc:5495] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b reported cstate change: term changed from 0 to 1, leader changed from <none> to bfcc372f4048444fb9454b78cb3c754b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "bfcc372f4048444fb9454b78cb3c754b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcc372f4048444fb9454b78cb3c754b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cf25f34585a4224aae4dd6355bdcc6b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a6b1160fa848019c342aeca364dc89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } health_report { overall_health: UNKNOWN } } }
I20260620 23:22:56.376348 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:56.371375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.0.97.2:43305)
0620 23:22:56.371549 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:56.371552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:56.371569 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:22:56.371727 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:56.371731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:56.371809 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:56.371931 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:22:56.372227 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.372948 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.374192 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.374640 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.374800 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.375260 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:22:56.375786 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:22:56.375888 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:56.376000 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 23:22:56.376085 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:22:56.384584 10863 raft_consensus.cc:1261] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b [term 1 FOLLOWER]: Refusing update from remote peer bfcc372f4048444fb9454b78cb3c754b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:56.384583 11130 raft_consensus.cc:1261] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89 [term 1 FOLLOWER]: Refusing update from remote peer bfcc372f4048444fb9454b78cb3c754b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:22:56.385532 11203 consensus_queue.cc:1035] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cf25f34585a4224aae4dd6355bdcc6b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:56.385907 11199 consensus_queue.cc:1035] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a6b1160fa848019c342aeca364dc89" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:22:56.396606 11206 mvcc.cc:205] Tried to move back new op lower bound from 7299062892060803072 to 7299062891773534208. Current Snapshot: MvccSnapshot[applied={T|T < 7299062892060803072}]
I20260620 23:22:56.403326 11208 mvcc.cc:205] Tried to move back new op lower bound from 7299062892060803072 to 7299062891773534208. Current Snapshot: MvccSnapshot[applied={T|T < 7299062892060803072}]
I20260620 23:22:56.411532 10719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:56.411825 10719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:22:56.416960 10719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbbf3f4b0984435fbf5507434b9ffccf: Sending DeleteTablet for 3 replicas of tablet cfcbdf4cfa18456395b1bc095ebeb522
I20260620 23:22:56.418371 10843 tablet_service.cc:1514] Processing DeleteTablet for tablet cfcbdf4cfa18456395b1bc095ebeb522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59332
I20260620 23:22:56.418375 10977 tablet_service.cc:1514] Processing DeleteTablet for tablet cfcbdf4cfa18456395b1bc095ebeb522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44192
I20260620 23:22:56.419377 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 10780
I20260620 23:22:56.419672 11110 tablet_service.cc:1514] Processing DeleteTablet for tablet cfcbdf4cfa18456395b1bc095ebeb522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32964
I20260620 23:22:56.420387 11109 tablet_service.cc:1514] Processing DeleteTablet for tablet cfcbdf4cfa18456395b1bc095ebeb522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:22:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32964
I20260620 23:22:56.436846 11217 tablet_replica.cc:330] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b: stopping tablet replica
I20260620 23:22:56.437335 11217 raft_consensus.cc:2229] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:22:56.438040 11217 raft_consensus.cc:2258] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:56.438248 11219 tablet_replica.cc:330] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89: stopping tablet replica
I20260620 23:22:56.438886 11219 raft_consensus.cc:2229] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:56.439515 11219 raft_consensus.cc:2258] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:56.440392 11218 tablet_replica.cc:330] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b: stopping tablet replica
I20260620 23:22:56.440770 11218 raft_consensus.cc:2229] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:22:56.441248 11218 raft_consensus.cc:2258] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:22:56.441597 11219 ts_tablet_manager.cc:1905] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:56.443864 11217 ts_tablet_manager.cc:1905] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 23:22:56 dist-test-slave-5kzz krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:22:56.447019 11218 ts_tablet_manager.cc:1905] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:22:56.448915 11219 ts_tablet_manager.cc:1918] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:56.449123 11219 log.cc:1196] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/cfcbdf4cfa18456395b1bc095ebeb522
I20260620 23:22:56.449802 11219 ts_tablet_manager.cc:1939] T cfcbdf4cfa18456395b1bc095ebeb522 P f6a6b1160fa848019c342aeca364dc89: Deleting consensus metadata
I20260620 23:22:56.451118 11217 ts_tablet_manager.cc:1918] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:56.451362 11217 log.cc:1196] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/cfcbdf4cfa18456395b1bc095ebeb522
I20260620 23:22:56.451675 10704 catalog_manager.cc:4841] TS f6a6b1160fa848019c342aeca364dc89 (127.0.97.3:37177): tablet cfcbdf4cfa18456395b1bc095ebeb522 (table test-table [id=49a3bd32cb7a4d2885630950d17a1e54]) successfully deleted
I20260620 23:22:56.452250 11217 ts_tablet_manager.cc:1939] T cfcbdf4cfa18456395b1bc095ebeb522 P bfcc372f4048444fb9454b78cb3c754b: Deleting consensus metadata
W20260620 23:22:56.452751 10704 catalog_manager.cc:4808] TS f6a6b1160fa848019c342aeca364dc89 (127.0.97.3:37177): delete failed for tablet cfcbdf4cfa18456395b1bc095ebeb522 because the tablet was not found. No further retry: Not found: Tablet not found: cfcbdf4cfa18456395b1bc095ebeb522
I20260620 23:22:56.453888 10704 catalog_manager.cc:4841] TS bfcc372f4048444fb9454b78cb3c754b (127.0.97.2:43305): tablet cfcbdf4cfa18456395b1bc095ebeb522 (table test-table [id=49a3bd32cb7a4d2885630950d17a1e54]) successfully deleted
I20260620 23:22:56.457463 11218 ts_tablet_manager.cc:1918] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:22:56.457680 11218 log.cc:1196] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/cfcbdf4cfa18456395b1bc095ebeb522
I20260620 23:22:56.458240 11218 ts_tablet_manager.cc:1939] T cfcbdf4cfa18456395b1bc095ebeb522 P 8cf25f34585a4224aae4dd6355bdcc6b: Deleting consensus metadata
I20260620 23:22:56.458751 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:56.421580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.0.97.1:41995)
0620 23:22:56.421743 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:56.421749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:56.431125 (+ 9376us) server_negotiation.cc:407] Connection header received
0620 23:22:56.431550 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:56.431555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:56.431664 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:56.431738 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:56.433370 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.434598 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.438291 (+ 3693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.438994 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.439089 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.451304 (+ 12215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:56.451316 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:56.451332 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:56.451379 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:56.455184 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:56.456065 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:56.456074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:56.456078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:56.456240 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:56.456714 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:56.456719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:56.456723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:56.457158 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:56.457429 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:56.458074 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 23:22:56.458321 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:22:56.460062 10903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:56.460192 10704 catalog_manager.cc:4841] TS 8cf25f34585a4224aae4dd6355bdcc6b (127.0.97.1:41995): tablet cfcbdf4cfa18456395b1bc095ebeb522 (table test-table [id=49a3bd32cb7a4d2885630950d17a1e54]) successfully deleted
I20260620 23:22:56.551960 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 10780
I20260620 23:22:56.569465 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 10914
Jun 20 23:22:56 dist-test-slave-5kzz krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:22:56.585829 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:56.570319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.0.97.2:43305)
0620 23:22:56.570531 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:56.570538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:56.570558 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:22:56.570652 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:56.570658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:56.570756 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:56.570876 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:56.572019 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.573200 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.574328 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.575148 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.575288 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.579370 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:56.579386 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:56.579406 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:56.579458 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:56.583151 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:56.583735 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:56.583744 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:56.583749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:56.583947 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:56.584257 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:56.584262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:56.584265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:56.584711 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:56.585043 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:56.585181 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 23:22:56.585405 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:22:56.586582 11037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:56.696683 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 10914
I20260620 23:22:56.712219 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 11047
Jun 20 23:22:56 dist-test-slave-5kzz krb5kdc[10674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:22:56.726140 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:22:56.712927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.0.97.3:37177)
0620 23:22:56.713081 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:22:56.713086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:22:56.713102 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:22:56.713236 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:22:56.713240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:22:56.713302 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:22:56.713382 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:22:56.714659 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.715408 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.716632 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:22:56.717015 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:22:56.717102 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:22:56.721328 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:22:56.721340 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:22:56.721356 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:22:56.721390 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:22:56.724043 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:56.724587 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:56.724593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:56.724595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:56.724707 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:22:56.724989 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:22:56.724992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:22:56.724994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:22:56.725286 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:22:56.725484 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 23:22:56.725727 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 23:22:56.725893 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:22:56.726863 11170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:56.812451 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 11047
I20260620 23:22:56.830561 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 10687
I20260620 23:22:56.831650 10749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:22:56.933279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 10687
2026-06-20T23:22:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4225 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:22:57 dist-test-slave-5kzz krb5kdc[11227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:22:57 dist-test-slave-5kzz krb5kdc[11227](info): set up 2 sockets
Jun 20 23:22:57 dist-test-slave-5kzz krb5kdc[11227](info): commencing operation
krb5kdc: starting...
W20260620 23:22:58.011468 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:22:58 dist-test-slave-5kzz krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:22:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:22:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 23:22:58.311621 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskaeh70X/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/postgres -l logfile start
2026-06-20 23:23:00.520 UTC [11257] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:23:00.520 UTC [11257] LOG: listening on IPv4 address "127.0.97.20", port 36129
2026-06-20 23:23:00.523 UTC [11257] LOG: listening on Unix socket "/tmp/.s.PGSQL.36129"
2026-06-20 23:23:00.545 UTC [11260] LOG: database system was shut down at 2026-06-20 23:22:59 UTC
2026-06-20 23:23:00.548 UTC [11257] LOG: database system is ready to accept connections
I20260620 23:23:01.499074 388 mini_postgres.cc:96] Postgres bound to 36129
2026-06-20 23:23:01.515 UTC [11268] FATAL: database "slave" does not exist
127.0.97.20:36129 - accepting connections
I20260620 23:23:01.516722 388 mini_ranger.cc:161] Starting Ranger...
I20260620 23:23:01.542147 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:23:01.940688 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:23:01.940793 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-admin
2026-06-20 23:23:02,243 [I] DB FLAVOR :POSTGRES
2026-06-20 23:23:02,243 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:23:02,243 [I] Checking connection..
2026-06-20 23:23:02,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:23:02,470 [I] Checking connection passed.
2026-06-20 23:23:02,471 [I] --------- Verifying version history table ---------
2026-06-20 23:23:02,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:23:02,734 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:23:02,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:23:02,985 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:23:02,985 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:23:02,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:23:03.239 UTC [11332] WARNING: there is no transaction in progress
2026-06-20 23:23:03,246 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:23:03,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:23:03,498 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:23:03,498 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:23:03,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:23:03,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:23:03,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5kzz.c.gcp-upstream.internal','N') ;"
2026-06-20 23:23:04,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:23:04,483 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:23:04,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:23:04,732 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:23:04,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:23:04,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:23:05,195 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:23:05,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:23:06.098 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.114 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.129 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.143 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.158 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.175 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.227 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.233 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.246 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.255 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.265 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.277 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.286 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.292 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.298 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06.304 UTC [11440] WARNING: there is no transaction in progress
2026-06-20 23:23:06,864 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:23:06,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:23:07,085 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:23:07,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:23:07,330 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:23:07,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:23:07,586 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:23:07,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:23:07,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:23:08,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5kzz.c.gcp-upstream.internal';"
2026-06-20 23:23:08,295 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:23:08,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:23:08,549 [I] DB_PATCHES have already been applied
I20260620 23:23:08.555136 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskaeh70X/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:23:08.555253 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 23:23:08.569591 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:41919
Jun 20, 2026 11:23:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:23:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:23:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:23:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:23:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:23:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:23:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:23:09.698376 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.182s user 0.000s sys 0.035s
Jun 20, 2026 11:23:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:23:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 11:23:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:23:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:23:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:23:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:23:31.028928 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:23:31 dist-test-slave-5kzz krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997811, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:23:31.190148 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:23:31.209272 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:23:31.936362 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:23:31.936548 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-kms
I20260620 23:23:31.936671 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:36129
2026-06-20 23:23:32,344 [I] DB FLAVOR :POSTGRES
2026-06-20 23:23:32,344 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:23:32,344 [I] Checking connection
2026-06-20 23:23:32,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:23:32,564 [I] connection success
2026-06-20 23:23:32,565 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:23:32,565 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:23:32,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:23:32,794 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:23:32,794 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:23:32,794 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:23:32,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36129/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:23:33,078 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:23:33.515488 388 mini_ranger_kms.cc:310] Created kms service
I20260620 23:23:33.702499 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:23:33.744381 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:23:33 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997811, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:23:34.120471 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:23:34.120779 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:23:34.120842 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 23:23:34.141649 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11620
I20260620 23:23:34.141825 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:33347
Jun 20, 2026 11:23:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:23:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:23:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:23:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:23:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:23:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 23:23:36.436875 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.247s user 0.121s sys 0.303s
I20260620 23:23:36.437121 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:23:36.437341 388 mini_ranger_kms.cc:401] 127.0.97.20:33347/kms/v1/keys
Jun 20 23:23:36 dist-test-slave-5kzz krb5kdc[11227](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997816, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:23:36 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997816, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 23:23:37 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997816, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 23:23:39 dist-test-slave-5kzz krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:23:39.679924 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40819
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40819
--ranger_config_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33347/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:23:40.024353 11671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:23:40.024617 11671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:23:40.024745 11671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:23:40.032109 11671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:23:40.032243 11671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:23:40.032325 11671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:23:40.032419 11671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:23:40.032500 11671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:23:40.042318 11671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:33347/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40819
--ranger_config_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:23:40.043586 11671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:23:40.044904 11671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:23:40.057170 11676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:23:40.057137 11677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:23:40.057124 11679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:23:40.057844 11671 server_base.cc:1034] running on GCE node
I20260620 23:23:40.058573 11671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:23:40.059916 11671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:23:40.061152 11671 hybrid_clock.cc:648] HybridClock initialized: now 1781997820061089 us; error 55 us; skew 500 ppm
I20260620 23:23:40.061475 11671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:23:40 dist-test-slave-5kzz krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:23:40.067976 11671 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:23:40.069315 11671 webserver.cc:460] Webserver started at http://127.0.97.62:45249/ using document root <none> and password file <none>
I20260620 23:23:40.069996 11671 fs_manager.cc:359] Metadata directory not provided
I20260620 23:23:40.070122 11671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:23:40.070405 11671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:23:40 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:23:40.129527 11671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "db523c4fcdf34e608ade18964a456c3f"
format_stamp: "Formatted at 2026-06-20 23:23:40 on dist-test-slave-5kzz"
server_key: "1e8d8e014a26940b3aa1fac85299b20a"
server_key_iv: "329c3af58f008c39d8ac2ff76bcd1462"
server_key_version: "kuduclusterkey@0"
I20260620 23:23:40.130468 11671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "db523c4fcdf34e608ade18964a456c3f"
format_stamp: "Formatted at 2026-06-20 23:23:40 on dist-test-slave-5kzz"
server_key: "1e8d8e014a26940b3aa1fac85299b20a"
server_key_iv: "329c3af58f008c39d8ac2ff76bcd1462"
server_key_version: "kuduclusterkey@0"
I20260620 23:23:40.135651 11671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:23:40.154815 11686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:23:40.156301 11671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:23:40.156476 11671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "db523c4fcdf34e608ade18964a456c3f"
format_stamp: "Formatted at 2026-06-20 23:23:40 on dist-test-slave-5kzz"
server_key: "1e8d8e014a26940b3aa1fac85299b20a"
server_key_iv: "329c3af58f008c39d8ac2ff76bcd1462"
server_key_version: "kuduclusterkey@0"
I20260620 23:23:40.156683 11671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:23:40.170331 11671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:23:40.174121 11671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:23:40.174417 11671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:23:40.194352 11671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40819
I20260620 23:23:40.194387 11737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40819 every 8 connection(s)
I20260620 23:23:40.195735 11671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:23:40.198391 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 11671
I20260620 23:23:40.198580 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:23:40.199330 388 external_mini_cluster.cc:1428] Setting key 34a7a42b600cbe21108bd0e278b39820
I20260620 23:23:40.199501 11738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:23:40.212854 11738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f: Bootstrap starting.
Jun 20 23:23:40 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:23:40.216190 11738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f: Neither blocks nor log segments found. Creating new log.
I20260620 23:23:40.217235 11738 log.cc:824] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:23:40.220573 11738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f: No bootstrap required, opened a new log
I20260620 23:23:40.223968 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:40.201141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.0.97.62:40819)
0620 23:23:40.204863 (+ 3722us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:40.204877 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:40.204915 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:23:40.206614 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:40.206635 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:40.207094 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:40.207485 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:23:40.208879 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:40.210260 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:40.211558 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:40.212210 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:40.212423 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:40.216459 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:23:40.216467 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:23:40.216480 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:23:40.216516 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:23:40.220052 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:40.220834 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:40.220840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:40.220846 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:40.220997 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:40.221396 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:40.221399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:40.221401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:40.221889 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:23:40.222083 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:40.222544 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 23:23:40.222965 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3495,"thread_start_us":3305,"threads_started":1}
I20260620 23:23:40.228978 11738 raft_consensus.cc:348] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db523c4fcdf34e608ade18964a456c3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40819 } }
I20260620 23:23:40.229312 11738 raft_consensus.cc:374] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:23:40.229425 11738 raft_consensus.cc:729] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db523c4fcdf34e608ade18964a456c3f, State: Initialized, Role: FOLLOWER
I20260620 23:23:40.229964 11738 consensus_queue.cc:260] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db523c4fcdf34e608ade18964a456c3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40819 } }
I20260620 23:23:40.230183 11738 raft_consensus.cc:388] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:23:40.230316 11738 raft_consensus.cc:482] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:23:40.230484 11738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:23:40.232656 11738 raft_consensus.cc:504] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db523c4fcdf34e608ade18964a456c3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40819 } }
I20260620 23:23:40.233124 11738 leader_election.cc:302] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db523c4fcdf34e608ade18964a456c3f; no voters:
I20260620 23:23:40.234028 11738 leader_election.cc:288] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:23:40.234125 11743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:23:40.234913 11743 raft_consensus.cc:686] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [term 1 LEADER]: Becoming Leader. State: Replica: db523c4fcdf34e608ade18964a456c3f, State: Running, Role: LEADER
I20260620 23:23:40.235484 11743 consensus_queue.cc:237] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db523c4fcdf34e608ade18964a456c3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40819 } }
I20260620 23:23:40.236124 11738 sys_catalog.cc:564] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:23:40.238188 11745 sys_catalog.cc:455] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader db523c4fcdf34e608ade18964a456c3f. Latest consensus state: current_term: 1 leader_uuid: "db523c4fcdf34e608ade18964a456c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db523c4fcdf34e608ade18964a456c3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40819 } } }
I20260620 23:23:40.238595 11745 sys_catalog.cc:458] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [sys.catalog]: This master's current role is: LEADER
I20260620 23:23:40.238983 11744 sys_catalog.cc:455] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db523c4fcdf34e608ade18964a456c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db523c4fcdf34e608ade18964a456c3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40819 } } }
I20260620 23:23:40.239390 11744 sys_catalog.cc:458] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f [sys.catalog]: This master's current role is: LEADER
I20260620 23:23:40.241755 11749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:23:40.247596 11749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:23:40.248229 11738 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:23:40.264678 11749 catalog_manager.cc:1347] Generated new cluster ID: 6dc3514b29d642d1ae3638659e99405b
I20260620 23:23:40.264904 11749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:23:40.274977 11749 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:23:40.276710 11749 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:23:40.282366 11749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f: Generated new TSK 0
I20260620 23:23:40.283241 11749 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:23:41.048359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40819
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:23:41 dist-test-slave-5kzz krb5kdc[11227](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 23:23:41.467643 11783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:23:41.468000 11783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:23:41.468210 11783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:23:41.479890 11783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:23:41.480022 11783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:23:41.480242 11783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:23:41.495044 11783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:33347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:23:41.496323 11783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:23:41.498031 11783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:23:41.513490 11793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:23:41.514235 11794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:23:41.527477 11796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:23:41.527453 11783 server_base.cc:1034] running on GCE node
I20260620 23:23:41.528393 11783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:23:41.529546 11783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:23:41.530789 11783 hybrid_clock.cc:648] HybridClock initialized: now 1781997821530715 us; error 53 us; skew 500 ppm
I20260620 23:23:41.531155 11783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:23:41 dist-test-slave-5kzz krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:23:41.535385 11783 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:23:41.540154 11783 webserver.cc:460] Webserver started at http://127.0.97.1:41063/ using document root <none> and password file <none>
I20260620 23:23:41.541169 11783 fs_manager.cc:359] Metadata directory not provided
I20260620 23:23:41.541311 11783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:23:41.541692 11783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:23:41 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:23:41.561013 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a"
format_stamp: "Formatted at 2026-06-20 23:23:41 on dist-test-slave-5kzz"
server_key: "50fddf3212f6fe39f06efc2a709eaeb2"
server_key_iv: "dbeed7b4e7a7ca3a45bff6d36787b7d9"
server_key_version: "kuduclusterkey@0"
I20260620 23:23:41.562206 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a"
format_stamp: "Formatted at 2026-06-20 23:23:41 on dist-test-slave-5kzz"
server_key: "50fddf3212f6fe39f06efc2a709eaeb2"
server_key_iv: "dbeed7b4e7a7ca3a45bff6d36787b7d9"
server_key_version: "kuduclusterkey@0"
I20260620 23:23:41.583750 11783 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.007s sys 0.012s
I20260620 23:23:41.625255 11803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:23:41.627086 11783 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.007s sys 0.004s
I20260620 23:23:41.627287 11783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a"
format_stamp: "Formatted at 2026-06-20 23:23:41 on dist-test-slave-5kzz"
server_key: "50fddf3212f6fe39f06efc2a709eaeb2"
server_key_iv: "dbeed7b4e7a7ca3a45bff6d36787b7d9"
server_key_version: "kuduclusterkey@0"
I20260620 23:23:41.627530 11783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:23:41.650426 11783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:23:41.655745 11783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:23:41.656121 11783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:23:41.661724 11783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:23:41.669840 11783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:23:41.669982 11783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:23:41.670115 11783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:23:41.670171 11783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:23:41.739748 11783 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36951
I20260620 23:23:41.739765 11915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36951 every 8 connection(s)
I20260620 23:23:41.741500 11783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:23:41.743968 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 11783
I20260620 23:23:41.744189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:23:41.744971 388 external_mini_cluster.cc:1428] Setting key 7ad7f51838dcd413da44d6005ab48498
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Jun 20 23:23:41 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:23:41.807195 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:41.748690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57841 (local address 127.0.97.62:40819)
0620 23:23:41.775111 (+ 26421us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:41.775120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:41.775153 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:23:41.783837 (+ 8684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:41.783842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:41.783950 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:41.784048 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:23:41.789209 (+ 5161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:41.790526 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:41.793294 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:41.794035 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:41.794143 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:41.798472 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:23:41.798481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:23:41.798488 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:23:41.798531 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:23:41.803060 (+ 4529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:41.804061 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:41.804069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:41.804075 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:41.804235 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:41.804842 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:41.804850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:41.804855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:41.805244 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:23:41.805382 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:41.806409 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0620 23:23:41.806664 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26244,"thread_start_us":33983,"threads_started":1}
I20260620 23:23:41.809844 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:41.747158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40819 (local address 127.0.97.1:57841)
0620 23:23:41.771112 (+ 23954us) negotiation.cc:107] Waiting for socket to connect
0620 23:23:41.771198 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 23:23:41.783527 (+ 12329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:23:41.787248 (+ 3721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:23:41.787263 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:23:41.788000 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:23:41.788967 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:41.788997 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:41.790812 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:41.790820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:41.793110 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:41.793125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:41.794492 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:41.794497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:41.794737 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:41.795486 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:23:41.795509 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:23:41.798226 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:23:41.803256 (+ 5030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:23:41.803268 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:23:41.803291 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:23:41.803923 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:23:41.804385 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:23:41.804392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:23:41.804398 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:23:41.804674 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:23:41.805487 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:23:41.805496 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:23:41.805978 (+ 482us) client_negotiation.cc:769] Sending connection context
0620 23:23:41.807749 (+ 1771us) client_negotiation.cc:240] Negotiation successful
0620 23:23:41.808346 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23464,"thread_start_us":27854,"threads_started":1}
I20260620 23:23:41.811951 11916 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40819
I20260620 23:23:41.812245 11916 heartbeater.cc:464] Registering TS with master...
I20260620 23:23:41.812923 11916 heartbeater.cc:511] Master 127.0.97.62:40819 requested a full tablet report, sending...
I20260620 23:23:41.815289 11703 ts_manager.cc:194] Registered new tserver with Master: 8cf4529e6e044bf1bd81cf7cc3e80e1a (127.0.97.1:36951)
I20260620 23:23:41.817484 11703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:23:41.832510 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40819
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:23:41 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 23:23:42 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 23:23:42.269297 11922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:23:42.269755 11922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:23:42.269948 11922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:23:42.281301 11922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:23:42.281486 11922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:23:42.281700 11922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:23:42.297216 11922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:33347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:23:42.298671 11922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:23:42.300458 11922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:23:42.347805 11929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:23:42.348910 11930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:23:42.356688 11922 server_base.cc:1034] running on GCE node
W20260620 23:23:42.356990 11932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:23:42.357693 11922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:23:42.358880 11922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:23:42.360138 11922 hybrid_clock.cc:648] HybridClock initialized: now 1781997822360072 us; error 55 us; skew 500 ppm
I20260620 23:23:42.360496 11922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:23:42 dist-test-slave-5kzz krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:23:42.363830 11922 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:23:42.379114 11922 webserver.cc:460] Webserver started at http://127.0.97.2:41519/ using document root <none> and password file <none>
I20260620 23:23:42.380215 11922 fs_manager.cc:359] Metadata directory not provided
I20260620 23:23:42.380424 11922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:23:42.380851 11922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:23:42 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:23:42.397377 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "779361ec5f8d4cb5b9d50bebfbfa5330"
format_stamp: "Formatted at 2026-06-20 23:23:42 on dist-test-slave-5kzz"
server_key: "aa3a0c3a18a7d1a26a2dd15e653f8c36"
server_key_iv: "0667cc83a15239767ea72d79fe031743"
server_key_version: "kuduclusterkey@0"
I20260620 23:23:42.398475 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "779361ec5f8d4cb5b9d50bebfbfa5330"
format_stamp: "Formatted at 2026-06-20 23:23:42 on dist-test-slave-5kzz"
server_key: "aa3a0c3a18a7d1a26a2dd15e653f8c36"
server_key_iv: "0667cc83a15239767ea72d79fe031743"
server_key_version: "kuduclusterkey@0"
I20260620 23:23:42.404618 11922 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 23:23:42.420979 11946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:23:42.423086 11922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:23:42.423257 11922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "779361ec5f8d4cb5b9d50bebfbfa5330"
format_stamp: "Formatted at 2026-06-20 23:23:42 on dist-test-slave-5kzz"
server_key: "aa3a0c3a18a7d1a26a2dd15e653f8c36"
server_key_iv: "0667cc83a15239767ea72d79fe031743"
server_key_version: "kuduclusterkey@0"
I20260620 23:23:42.423456 11922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:23:42.445326 11922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:23:42.449231 11922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:23:42.449522 11922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:23:42.450439 11922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:23:42.451929 11922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:23:42.452001 11922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:23:42.452147 11922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:23:42.452194 11922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:23:42.490114 11922 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39369
I20260620 23:23:42.490132 12058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39369 every 8 connection(s)
I20260620 23:23:42.491384 11922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:23:42.495034 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 11922
I20260620 23:23:42.495208 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:23:42.495606 388 external_mini_cluster.cc:1428] Setting key 80102610328dfb884007fb744f15a61c
Jun 20 23:23:42 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 23:23:42.522013 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:42.494285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59491 (local address 127.0.97.62:40819)
0620 23:23:42.494800 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:42.494807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:42.502076 (+ 7269us) server_negotiation.cc:407] Connection header received
0620 23:23:42.504660 (+ 2584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:42.504665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:42.504771 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:42.504897 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:23:42.506267 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:42.507536 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:42.510203 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:42.511037 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:42.511184 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:42.515503 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:23:42.515509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:23:42.515514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:23:42.515553 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:23:42.518910 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:42.519676 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:42.519681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:42.519684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:42.519830 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:42.520182 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:42.520187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:42.520190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:42.520512 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:23:42.520639 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:42.521444 (+ 805us) server_negotiation.cc:299] Negotiation successful
0620 23:23:42.521656 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":266,"threads_started":1}
I20260620 23:23:42.523097 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:42.494937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40819 (local address 127.0.97.2:59491)
0620 23:23:42.501908 (+ 6971us) negotiation.cc:107] Waiting for socket to connect
0620 23:23:42.501962 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:23:42.504395 (+ 2433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:23:42.505128 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:23:42.505136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:23:42.505529 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:23:42.506110 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:42.506127 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:42.507716 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:42.507722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:42.510052 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:42.510067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:42.511265 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:42.511272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:42.511553 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:42.512368 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:23:42.512420 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:23:42.515275 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:23:42.519031 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:23:42.519037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:23:42.519051 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:23:42.519560 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:23:42.519930 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:23:42.519933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:23:42.519935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:23:42.520091 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:23:42.520656 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:23:42.520664 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:23:42.521094 (+ 430us) client_negotiation.cc:769] Sending connection context
0620 23:23:42.521356 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 23:23:42.521850 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6656,"thread_start_us":6438,"threads_started":1}
I20260620 23:23:42.524468 12059 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40819
I20260620 23:23:42.524771 12059 heartbeater.cc:464] Registering TS with master...
I20260620 23:23:42.525435 12059 heartbeater.cc:511] Master 127.0.97.62:40819 requested a full tablet report, sending...
I20260620 23:23:42.526906 11703 ts_manager.cc:194] Registered new tserver with Master: 779361ec5f8d4cb5b9d50bebfbfa5330 (127.0.97.2:39369)
I20260620 23:23:42.527938 11703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:23:42.551855 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40819
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:23:42.821679 11916 heartbeater.cc:503] Master 127.0.97.62:40819 was elected leader, sending a full tablet report...
W20260620 23:23:42.890594 12065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:23:42.890904 12065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:23:42.891079 12065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:23:42.897980 12065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:23:42.898074 12065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:23:42.898227 12065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:23:42.907910 12065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:33347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:23:42.909062 12065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:23:42.910215 12065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:23:42.922760 12070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:23:42.922811 12073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:23:42.922839 12071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:23:42.923537 12065 server_base.cc:1034] running on GCE node
I20260620 23:23:42.924175 12065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:23:42.925454 12065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:23:42.926661 12065 hybrid_clock.cc:648] HybridClock initialized: now 1781997822926601 us; error 50 us; skew 500 ppm
I20260620 23:23:42.926990 12065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:23:42 dist-test-slave-5kzz krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:23:42.929836 12065 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:23:42.931293 12065 webserver.cc:460] Webserver started at http://127.0.97.3:39931/ using document root <none> and password file <none>
I20260620 23:23:42.931910 12065 fs_manager.cc:359] Metadata directory not provided
I20260620 23:23:42.932039 12065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:23:42.932327 12065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:23:42 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:23:42.948251 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "198b22e7e8ff497fbaaaf583ce4a66d7"
format_stamp: "Formatted at 2026-06-20 23:23:42 on dist-test-slave-5kzz"
server_key: "d74c36f5c0de8cb5382bd7ee79b85fb6"
server_key_iv: "eba93678417eb839b7173dfec3410c03"
server_key_version: "kuduclusterkey@0"
I20260620 23:23:42.949047 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "198b22e7e8ff497fbaaaf583ce4a66d7"
format_stamp: "Formatted at 2026-06-20 23:23:42 on dist-test-slave-5kzz"
server_key: "d74c36f5c0de8cb5382bd7ee79b85fb6"
server_key_iv: "eba93678417eb839b7173dfec3410c03"
server_key_version: "kuduclusterkey@0"
I20260620 23:23:42.953985 12065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:23:42.967449 12080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:23:42.969012 12065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:23:42.969188 12065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "198b22e7e8ff497fbaaaf583ce4a66d7"
format_stamp: "Formatted at 2026-06-20 23:23:42 on dist-test-slave-5kzz"
server_key: "d74c36f5c0de8cb5382bd7ee79b85fb6"
server_key_iv: "eba93678417eb839b7173dfec3410c03"
server_key_version: "kuduclusterkey@0"
I20260620 23:23:42.969457 12065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:23:42.992017 12065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:23:42.995819 12065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:23:42.996132 12065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:23:42.997125 12065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:23:42.998481 12065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:23:42.998555 12065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:23:42.998746 12065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:23:42.998804 12065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 23:23:43.033268 12065 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33541
I20260620 23:23:43.033710 12192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33541 every 8 connection(s)
I20260620 23:23:43.034976 12065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:23:43.036010 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 12065
I20260620 23:23:43.036175 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:23:43.036689 388 external_mini_cluster.cc:1428] Setting key fd661cdfeaf4a69f1201fdc45392759c
Jun 20 23:23:43 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:23:43.054050 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.037753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49711 (local address 127.0.97.62:40819)
0620 23:23:43.038348 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.038355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.039427 (+ 1072us) server_negotiation.cc:407] Connection header received
0620 23:23:43.041150 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.041154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.041219 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.041316 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:23:43.042529 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.043346 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.045029 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.045432 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.045536 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.048686 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:23:43.048691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:23:43.048694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:23:43.048733 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:23:43.051565 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:43.052339 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:43.052343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:43.052345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:43.052454 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:43.052766 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:43.052769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:43.052771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:43.053003 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:23:43.053130 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.053654 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.053806 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":342,"threads_started":1}
I20260620 23:23:43.055392 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.038419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40819 (local address 127.0.97.3:49711)
0620 23:23:43.039276 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 23:23:43.039331 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:23:43.040917 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:23:43.041466 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:23:43.041473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:23:43.041868 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:23:43.042380 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.042396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.043467 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.043470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.044861 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.044881 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.045630 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.045635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.045829 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.046478 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:23:43.046503 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:23:43.048488 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:23:43.051690 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:23:43.051698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:23:43.051713 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:23:43.052246 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:23:43.052539 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:23:43.052542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:23:43.052544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:23:43.052688 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:23:43.053112 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:23:43.053117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:23:43.053379 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 23:23:43.053591 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 23:23:43.053917 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":385,"threads_started":1}
I20260620 23:23:43.056582 12193 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40819
I20260620 23:23:43.056856 12193 heartbeater.cc:464] Registering TS with master...
I20260620 23:23:43.057473 12193 heartbeater.cc:511] Master 127.0.97.62:40819 requested a full tablet report, sending...
I20260620 23:23:43.058736 11703 ts_manager.cc:194] Registered new tserver with Master: 198b22e7e8ff497fbaaaf583ce4a66d7 (127.0.97.3:33541)
I20260620 23:23:43.059419 11703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49711
I20260620 23:23:43.063925 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:23:43.079195 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.068035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.0.97.62:40819)
0620 23:23:43.068204 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.068212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.068679 (+ 467us) server_negotiation.cc:407] Connection header received
0620 23:23:43.068795 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.068798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.068867 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.068943 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:23:43.070161 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.070882 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.072086 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.072498 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.072583 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.074880 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:23:43.074898 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:23:43.074905 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:23:43.074939 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:23:43.077150 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:43.077679 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:43.077682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:43.077685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:43.077802 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:43.078090 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:43.078092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:43.078094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:43.078315 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:23:43.078472 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.078749 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.078890 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:23:43.083088 11703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:23:43.089715 11703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:23:43.111985 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.100991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.0.97.2:39369)
0620 23:23:43.103850 (+ 2859us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.103861 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.103891 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:23:43.103993 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.104002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.104291 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.105178 (+ 887us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:23:43.105417 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.107008 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.109875 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.110801 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.110927 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.111130 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.111475 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.111636 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2697,"thread_start_us":2592,"threads_started":1}
I20260620 23:23:43.112641 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.100775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39369 (local address 127.0.0.1:43722)
0620 23:23:43.103037 (+ 2262us) negotiation.cc:107] Waiting for socket to connect
0620 23:23:43.103091 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:23:43.103293 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:23:43.104549 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:23:43.104556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:23:43.104585 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:23:43.105007 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.105019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.107141 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.107148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.109737 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.109747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.110987 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.110993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.111187 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.111285 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 23:23:43.111420 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 23:23:43.111940 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1802,"mutex_wait_us":57,"thread_start_us":3599,"threads_started":1}
I20260620 23:23:43.113636 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.102165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48356 (local address 127.0.97.1:36951)
0620 23:23:43.104257 (+ 2092us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.104265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.104740 (+ 475us) server_negotiation.cc:407] Connection header received
0620 23:23:43.105011 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.105018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.105235 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.105402 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:23:43.106355 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.107877 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.110905 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.112271 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.112380 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.112593 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.113168 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.113356 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1943,"thread_start_us":1793,"threads_started":1}
I20260620 23:23:43.114315 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.101460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36951 (local address 127.0.0.1:48356)
0620 23:23:43.104631 (+ 3171us) negotiation.cc:107] Waiting for socket to connect
0620 23:23:43.104651 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:23:43.104825 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:23:43.105650 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:23:43.105654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:23:43.105677 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:23:43.106113 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.106125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.108006 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.108011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.110737 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.110753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.112793 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.112798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.112975 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.113006 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:23:43.113942 (+ 936us) client_negotiation.cc:240] Negotiation successful
0620 23:23:43.114064 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3060,"mutex_wait_us":20,"thread_start_us":947,"threads_started":1}
I20260620 23:23:43.115345 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.105713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33541 (local address 127.0.0.1:50356)
0620 23:23:43.106368 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 23:23:43.106385 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:23:43.106513 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:23:43.107081 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:23:43.107085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:23:43.107109 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:23:43.108966 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.108979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.110815 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.110820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.112743 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.112761 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.114600 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.114608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.114786 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.114819 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:23:43.114914 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 23:23:43.115036 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":6,"thread_start_us":379,"threads_started":1}
I20260620 23:23:43.115442 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.105052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.0.97.3:33541)
0620 23:23:43.105715 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.105729 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.106486 (+ 757us) server_negotiation.cc:407] Connection header received
0620 23:23:43.106647 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.106655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.106881 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.107071 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:23:43.109189 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.110670 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.112921 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.114120 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.114214 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.114427 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.114962 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.115112 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":378,"threads_started":1}
I20260620 23:23:43.117379 11994 tablet_service.cc:1467] Processing CreateTablet for tablet 75ad7e3a0dc442e7916936761420902a (DEFAULT_TABLE table=test-table [id=5c0a8e7d630a4165a9841de25dec39ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:23:43.117573 11851 tablet_service.cc:1467] Processing CreateTablet for tablet 75ad7e3a0dc442e7916936761420902a (DEFAULT_TABLE table=test-table [id=5c0a8e7d630a4165a9841de25dec39ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:23:43.118855 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ad7e3a0dc442e7916936761420902a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:23:43.118930 11851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ad7e3a0dc442e7916936761420902a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:23:43.118880 12128 tablet_service.cc:1467] Processing CreateTablet for tablet 75ad7e3a0dc442e7916936761420902a (DEFAULT_TABLE table=test-table [id=5c0a8e7d630a4165a9841de25dec39ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:23:43.120048 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ad7e3a0dc442e7916936761420902a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:23:43.129572 12212 tablet_bootstrap.cc:492] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330: Bootstrap starting.
I20260620 23:23:43.133203 12213 tablet_bootstrap.cc:492] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a: Bootstrap starting.
I20260620 23:23:43.133603 12212 tablet_bootstrap.cc:654] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330: Neither blocks nor log segments found. Creating new log.
I20260620 23:23:43.135373 12212 log.cc:824] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330: Log is configured to *not* fsync() on all Append() calls
I20260620 23:23:43.137281 12213 tablet_bootstrap.cc:654] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a: Neither blocks nor log segments found. Creating new log.
I20260620 23:23:43.138453 12213 log.cc:824] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:23:43.138787 12214 tablet_bootstrap.cc:492] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7: Bootstrap starting.
I20260620 23:23:43.139317 12212 tablet_bootstrap.cc:492] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330: No bootstrap required, opened a new log
I20260620 23:23:43.139688 12212 ts_tablet_manager.cc:1397] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260620 23:23:43.141469 12214 tablet_bootstrap.cc:654] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7: Neither blocks nor log segments found. Creating new log.
I20260620 23:23:43.142426 12214 log.cc:824] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:23:43.145704 12213 tablet_bootstrap.cc:492] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a: No bootstrap required, opened a new log
I20260620 23:23:43.145975 12213 ts_tablet_manager.cc:1397] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260620 23:23:43.147723 12214 tablet_bootstrap.cc:492] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7: No bootstrap required, opened a new log
I20260620 23:23:43.147953 12214 ts_tablet_manager.cc:1397] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 23:23:43.150629 12212 raft_consensus.cc:348] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39369 } } peers { permanent_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33541 } } peers { permanent_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36951 } }
I20260620 23:23:43.151173 12212 raft_consensus.cc:374] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:23:43.151327 12212 raft_consensus.cc:729] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 779361ec5f8d4cb5b9d50bebfbfa5330, State: Initialized, Role: FOLLOWER
I20260620 23:23:43.152014 12212 consensus_queue.cc:260] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39369 } } peers { permanent_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33541 } } peers { permanent_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36951 } }
I20260620 23:23:43.153069 12059 heartbeater.cc:503] Master 127.0.97.62:40819 was elected leader, sending a full tablet report...
I20260620 23:23:43.153684 12212 ts_tablet_manager.cc:1428] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260620 23:23:43.156524 12213 raft_consensus.cc:348] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39369 } } peers { permanent_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33541 } } peers { permanent_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36951 } }
I20260620 23:23:43.156553 12214 raft_consensus.cc:348] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39369 } } peers { permanent_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33541 } } peers { permanent_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36951 } }
I20260620 23:23:43.156865 12213 raft_consensus.cc:374] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:23:43.156971 12214 raft_consensus.cc:374] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:23:43.156998 12213 raft_consensus.cc:729] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf4529e6e044bf1bd81cf7cc3e80e1a, State: Initialized, Role: FOLLOWER
I20260620 23:23:43.157085 12214 raft_consensus.cc:729] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198b22e7e8ff497fbaaaf583ce4a66d7, State: Initialized, Role: FOLLOWER
I20260620 23:23:43.157632 12213 consensus_queue.cc:260] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39369 } } peers { permanent_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33541 } } peers { permanent_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36951 } }
I20260620 23:23:43.157688 12214 consensus_queue.cc:260] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39369 } } peers { permanent_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33541 } } peers { permanent_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36951 } }
I20260620 23:23:43.159303 12213 ts_tablet_manager.cc:1428] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 23:23:43.160151 12193 heartbeater.cc:503] Master 127.0.97.62:40819 was elected leader, sending a full tablet report...
I20260620 23:23:43.161197 12214 ts_tablet_manager.cc:1428] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
W20260620 23:23:43.243705 12060 tablet.cc:2369] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:23:43.262126 12220 raft_consensus.cc:482] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:23:43.262373 12220 raft_consensus.cc:504] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39369 } } peers { permanent_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33541 } } peers { permanent_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36951 } }
I20260620 23:23:43.264041 12220 leader_election.cc:288] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 779361ec5f8d4cb5b9d50bebfbfa5330 (127.0.97.2:39369), 8cf4529e6e044bf1bd81cf7cc3e80e1a (127.0.97.1:36951)
I20260620 23:23:43.264070 12219 raft_consensus.cc:482] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:23:43.264415 12219 raft_consensus.cc:504] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39369 } } peers { permanent_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33541 } } peers { permanent_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36951 } }
I20260620 23:23:43.267105 12219 leader_election.cc:288] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 779361ec5f8d4cb5b9d50bebfbfa5330 (127.0.97.2:39369), 198b22e7e8ff497fbaaaf583ce4a66d7 (127.0.97.3:33541)
I20260620 23:23:43.273726 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.264832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41921 (local address 127.0.97.1:36951)
0620 23:23:43.265000 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.265004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.265043 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:23:43.265127 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.265132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.265218 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.265316 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:23:43.265962 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.267137 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.270076 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.271743 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.271906 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.272040 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.273259 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.273432 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:23:43.273135 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.264329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36951 (local address 127.0.97.3:41921)
0620 23:23:43.264479 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:23:43.264498 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:23:43.264616 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:23:43.265517 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:23:43.265522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:23:43.265549 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:23:43.265752 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.265761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.267278 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.267286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.269846 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.269865 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.272132 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.272140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.272370 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.272418 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 23:23:43.272514 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:23:43.272709 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":1}
I20260620 23:23:43.274238 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.264487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46811 (local address 127.0.97.2:39369)
0620 23:23:43.264684 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.264692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.266050 (+ 1358us) server_negotiation.cc:407] Connection header received
0620 23:23:43.266134 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.266139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.266207 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.266301 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:23:43.266883 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.267957 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.271040 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.272627 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.272784 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.272920 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.273633 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.273846 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:23:43.275164 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.264922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39369 (local address 127.0.97.3:46811)
0620 23:23:43.265613 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 23:23:43.265632 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:23:43.265741 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:23:43.266337 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:23:43.266341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:23:43.266367 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:23:43.266705 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.266716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.268184 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.268192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.270836 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.270854 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.273148 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.273157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.273380 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.273428 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 23:23:43.274667 (+ 1239us) client_negotiation.cc:240] Negotiation successful
0620 23:23:43.274836 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":394,"threads_started":1}
I20260620 23:23:43.275784 11871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ad7e3a0dc442e7916936761420902a" candidate_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" is_pre_election: true
I20260620 23:23:43.276095 12014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ad7e3a0dc442e7916936761420902a" candidate_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" is_pre_election: true
I20260620 23:23:43.276275 11871 raft_consensus.cc:2454] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 198b22e7e8ff497fbaaaf583ce4a66d7 in term 0.
I20260620 23:23:43.276628 12014 raft_consensus.cc:2454] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 198b22e7e8ff497fbaaaf583ce4a66d7 in term 0.
I20260620 23:23:43.277112 12082 leader_election.cc:302] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198b22e7e8ff497fbaaaf583ce4a66d7, 8cf4529e6e044bf1bd81cf7cc3e80e1a; no voters:
I20260620 23:23:43.277848 12220 raft_consensus.cc:2781] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:23:43.278065 12220 raft_consensus.cc:482] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:23:43.278194 12220 raft_consensus.cc:3037] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:23:43.281459 12220 raft_consensus.cc:504] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39369 } } peers { permanent_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33541 } } peers { permanent_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36951 } }
I20260620 23:23:43.282354 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.267573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34341 (local address 127.0.97.2:39369)
0620 23:23:43.274530 (+ 6957us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.274538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.276009 (+ 1471us) server_negotiation.cc:407] Connection header received
0620 23:23:43.276267 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.276274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.276364 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.276429 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:23:43.277209 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.278646 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.280312 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.281488 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.281611 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.281709 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.281973 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.282090 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6812,"thread_start_us":7331,"threads_started":1}
I20260620 23:23:43.282377 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.267188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39369 (local address 127.0.97.1:34341)
0620 23:23:43.275905 (+ 8717us) negotiation.cc:107] Waiting for socket to connect
0620 23:23:43.275929 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:23:43.276109 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:23:43.276599 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:23:43.276603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:23:43.276629 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:23:43.277048 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.277061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.278828 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.278835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.280173 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.280186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.281666 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.281672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.281829 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.281864 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 23:23:43.281954 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:23:43.282067 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8577,"thread_start_us":7984,"threads_started":1}
I20260620 23:23:43.282550 12014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ad7e3a0dc442e7916936761420902a" candidate_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330"
I20260620 23:23:43.282769 12014 raft_consensus.cc:3037] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:23:43.283210 12013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ad7e3a0dc442e7916936761420902a" candidate_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" is_pre_election: true
I20260620 23:23:43.283546 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.276169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46115 (local address 127.0.97.3:33541)
0620 23:23:43.276330 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.276337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.277518 (+ 1181us) server_negotiation.cc:407] Connection header received
0620 23:23:43.277735 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.277742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.277821 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.277915 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:23:43.278930 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.279971 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.281478 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.282703 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.282826 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.282913 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.283229 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.283348 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:23:43.283593 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.276470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33541 (local address 127.0.97.1:46115)
0620 23:23:43.277417 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0620 23:23:43.277434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:23:43.277600 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:23:43.277950 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:23:43.277954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:23:43.277979 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:23:43.278753 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.278764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.280116 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.280122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.281323 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:23:43.281333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.282869 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.282875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:23:43.283043 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.283084 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:23:43.283174 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:23:43.283299 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":977,"threads_started":1}
I20260620 23:23:43.284787 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ad7e3a0dc442e7916936761420902a" candidate_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" is_pre_election: true
W20260620 23:23:43.284574 11917 tablet.cc:2369] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:23:43.285085 11871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ad7e3a0dc442e7916936761420902a" candidate_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a"
I20260620 23:23:43.285224 12148 raft_consensus.cc:2379] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8cf4529e6e044bf1bd81cf7cc3e80e1a in current term 1: Already voted for candidate 198b22e7e8ff497fbaaaf583ce4a66d7 in this term.
I20260620 23:23:43.285279 11871 raft_consensus.cc:3037] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:23:43.286244 12014 raft_consensus.cc:2454] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 198b22e7e8ff497fbaaaf583ce4a66d7 in term 1.
I20260620 23:23:43.286664 12084 leader_election.cc:302] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198b22e7e8ff497fbaaaf583ce4a66d7, 779361ec5f8d4cb5b9d50bebfbfa5330; no voters:
I20260620 23:23:43.286695 11807 leader_election.cc:302] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8cf4529e6e044bf1bd81cf7cc3e80e1a; no voters: 198b22e7e8ff497fbaaaf583ce4a66d7, 779361ec5f8d4cb5b9d50bebfbfa5330
W20260620 23:23:43.286823 12194 tablet.cc:2369] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:23:43.287875 11871 raft_consensus.cc:2454] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 198b22e7e8ff497fbaaaf583ce4a66d7 in term 1.
I20260620 23:23:43.288046 12219 raft_consensus.cc:2726] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 23:23:43.288154 12220 leader_election.cc:288] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [CANDIDATE]: Term 1 election: Requested vote from peers 779361ec5f8d4cb5b9d50bebfbfa5330 (127.0.97.2:39369), 8cf4529e6e044bf1bd81cf7cc3e80e1a (127.0.97.1:36951)
I20260620 23:23:43.288331 12220 raft_consensus.cc:2781] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:23:43.288465 12220 raft_consensus.cc:686] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 1 LEADER]: Becoming Leader. State: Replica: 198b22e7e8ff497fbaaaf583ce4a66d7, State: Running, Role: LEADER
I20260620 23:23:43.288982 12220 consensus_queue.cc:237] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39369 } } peers { permanent_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33541 } } peers { permanent_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36951 } }
I20260620 23:23:43.292876 11703 catalog_manager.cc:5495] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 198b22e7e8ff497fbaaaf583ce4a66d7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "198b22e7e8ff497fbaaaf583ce4a66d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36951 } health_report { overall_health: UNKNOWN } } }
I20260620 23:23:43.347465 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.342511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.0.97.3:33541)
0620 23:23:43.342690 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.342696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.342768 (+ 72us) server_negotiation.cc:407] Connection header received
0620 23:23:43.342954 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.342958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.343036 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.343150 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:23:43.343441 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.344194 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.345513 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.345916 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.346005 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.346381 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:23:43.346944 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:23:43.347047 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.347105 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.347161 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:23:43.355444 12014 raft_consensus.cc:1261] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330 [term 1 FOLLOWER]: Refusing update from remote peer 198b22e7e8ff497fbaaaf583ce4a66d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:23:43.355499 11871 raft_consensus.cc:1261] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [term 1 FOLLOWER]: Refusing update from remote peer 198b22e7e8ff497fbaaaf583ce4a66d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:23:43.356289 12225 consensus_queue.cc:1035] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "779361ec5f8d4cb5b9d50bebfbfa5330" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:23:43.356627 12220 consensus_queue.cc:1035] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cf4529e6e044bf1bd81cf7cc3e80e1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:23:43.381531 11702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:23:43.381804 11702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:23:43.384893 11702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db523c4fcdf34e608ade18964a456c3f: Sending DeleteTablet for 3 replicas of tablet 75ad7e3a0dc442e7916936761420902a
I20260620 23:23:43.386065 11851 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ad7e3a0dc442e7916936761420902a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:23:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48356
I20260620 23:23:43.386154 11994 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ad7e3a0dc442e7916936761420902a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:23:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43722
I20260620 23:23:43.386234 12128 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ad7e3a0dc442e7916936761420902a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:23:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50356
I20260620 23:23:43.388597 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 11783
I20260620 23:23:43.415897 12238 tablet_replica.cc:330] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330: stopping tablet replica
I20260620 23:23:43.416610 12239 tablet_replica.cc:330] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a: stopping tablet replica
I20260620 23:23:43.416905 12240 tablet_replica.cc:330] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7: stopping tablet replica
I20260620 23:23:43.417343 12238 raft_consensus.cc:2229] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:23:43.417343 12240 raft_consensus.cc:2229] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:23:43.417734 12239 raft_consensus.cc:2229] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:23:43.418037 12240 raft_consensus.cc:2258] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:23:43.418540 12239 raft_consensus.cc:2258] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:23:43.418856 12238 raft_consensus.cc:2258] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:23:43.420444 12240 ts_tablet_manager.cc:1905] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:23:43.421033 12238 ts_tablet_manager.cc:1905] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:23:43.422076 12239 ts_tablet_manager.cc:1905] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:23:43.427637 12238 ts_tablet_manager.cc:1918] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:23:43.427867 12238 log.cc:1196] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/75ad7e3a0dc442e7916936761420902a
I20260620 23:23:43.427856 12240 ts_tablet_manager.cc:1918] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:23:43.428011 12240 log.cc:1196] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/75ad7e3a0dc442e7916936761420902a
I20260620 23:23:43.428395 12238 ts_tablet_manager.cc:1939] T 75ad7e3a0dc442e7916936761420902a P 779361ec5f8d4cb5b9d50bebfbfa5330: Deleting consensus metadata
I20260620 23:23:43.428480 12240 ts_tablet_manager.cc:1939] T 75ad7e3a0dc442e7916936761420902a P 198b22e7e8ff497fbaaaf583ce4a66d7: Deleting consensus metadata
I20260620 23:23:43.430127 11690 catalog_manager.cc:4841] TS 198b22e7e8ff497fbaaaf583ce4a66d7 (127.0.97.3:33541): tablet 75ad7e3a0dc442e7916936761420902a (table test-table [id=5c0a8e7d630a4165a9841de25dec39ed]) successfully deleted
Jun 20 23:23:43 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:23:43.430526 11690 catalog_manager.cc:4841] TS 779361ec5f8d4cb5b9d50bebfbfa5330 (127.0.97.2:39369): tablet 75ad7e3a0dc442e7916936761420902a (table test-table [id=5c0a8e7d630a4165a9841de25dec39ed]) successfully deleted
I20260620 23:23:43.435143 12239 ts_tablet_manager.cc:1918] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:23:43.435292 12239 log.cc:1196] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/75ad7e3a0dc442e7916936761420902a
I20260620 23:23:43.435844 12239 ts_tablet_manager.cc:1939] T 75ad7e3a0dc442e7916936761420902a P 8cf4529e6e044bf1bd81cf7cc3e80e1a: Deleting consensus metadata
I20260620 23:23:43.437502 11688 catalog_manager.cc:4841] TS 8cf4529e6e044bf1bd81cf7cc3e80e1a (127.0.97.1:36951): tablet 75ad7e3a0dc442e7916936761420902a (table test-table [id=5c0a8e7d630a4165a9841de25dec39ed]) successfully deleted
I20260620 23:23:43.439308 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.389897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.0.97.1:36951)
0620 23:23:43.390063 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.390070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.414042 (+ 23972us) server_negotiation.cc:407] Connection header received
0620 23:23:43.414567 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.414573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.414711 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.414843 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:23:43.416803 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.417973 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.422577 (+ 4604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.423193 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.423295 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.431864 (+ 8569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:23:43.431882 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:23:43.431899 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:23:43.431970 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:23:43.436268 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:43.437191 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:43.437200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:43.437205 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:43.437448 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:43.437890 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:43.437893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:43.437895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:43.438234 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:23:43.438444 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.438874 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.439022 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:23:43.439906 11911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:23:43.529052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 11783
I20260620 23:23:43.551620 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 11922
Jun 20 23:23:43 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:23:43.567965 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.552357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.0.97.2:39369)
0620 23:23:43.552519 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.552524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.552566 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:23:43.552755 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.552759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.552844 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.552923 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:23:43.554572 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.555357 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.557292 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.557662 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.557747 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.562526 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:23:43.562538 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:23:43.562554 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:23:43.562587 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:23:43.565281 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:43.565971 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:43.565978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:43.565980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:43.566106 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:43.566457 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:43.566464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:43.566467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:43.566828 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:23:43.567117 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.567401 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.567668 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:23:43.568513 12054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:23:43.663199 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 11922
I20260620 23:23:43.679659 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 12065
Jun 20 23:23:43 dist-test-slave-5kzz krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:23:43.696489 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:23:43.680493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.0.97.3:33541)
0620 23:23:43.680682 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:23:43.680688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:23:43.680709 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:23:43.680781 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:23:43.680787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:23:43.680896 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:23:43.681021 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:23:43.682228 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.683359 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.684725 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:23:43.685417 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:23:43.685540 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:23:43.689775 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:23:43.689793 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:23:43.689821 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:23:43.689862 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:23:43.693693 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:43.694412 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:43.694421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:43.694426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:43.694565 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:23:43.694947 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:23:43.694952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:23:43.694956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:23:43.695422 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:23:43.695699 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 23:23:43.695917 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:23:43.696129 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:23:43.697353 12188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:23:43.814043 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 12065
I20260620 23:23:43.832410 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 11671
I20260620 23:23:43.833526 11733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:23:43.939409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 11671
I20260620 23:23:43.960963 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:23:44.296950 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:23:44.297199 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:23:44.643725 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:23:44.644 UTC [11257] LOG: received smart shutdown request
2026-06-20 23:23:44.646 UTC [11257] LOG: background worker "logical replication launcher" (PID 11266) exited with exit code 1
2026-06-20 23:23:44.646 UTC [11261] LOG: shutting down
2026-06-20 23:23:44.781 UTC [11257] LOG: database system is shut down
2026-06-20T23:23:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47968 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:23:44 dist-test-slave-5kzz krb5kdc[12254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:23:44 dist-test-slave-5kzz krb5kdc[12254](info): set up 2 sockets
Jun 20 23:23:44 dist-test-slave-5kzz krb5kdc[12254](info): commencing operation
krb5kdc: starting...
W20260620 23:23:45.983016 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:23:46 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:23:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:23:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 23:23:46.288655 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskaeh70X/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/postgres -l logfile start
2026-06-20 23:23:48.504 UTC [12284] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:23:48.504 UTC [12284] LOG: listening on IPv4 address "127.0.97.20", port 56695
2026-06-20 23:23:48.507 UTC [12284] LOG: listening on Unix socket "/tmp/.s.PGSQL.56695"
2026-06-20 23:23:48.528 UTC [12287] LOG: database system was shut down at 2026-06-20 23:23:47 UTC
2026-06-20 23:23:48.532 UTC [12284] LOG: database system is ready to accept connections
I20260620 23:23:49.486845 388 mini_postgres.cc:96] Postgres bound to 56695
2026-06-20 23:23:49.503 UTC [12295] FATAL: database "slave" does not exist
127.0.97.20:56695 - accepting connections
I20260620 23:23:49.503860 388 mini_ranger.cc:161] Starting Ranger...
I20260620 23:23:49.527721 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:23:49.942143 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:23:49.942258 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-admin
2026-06-20 23:23:50,216 [I] DB FLAVOR :POSTGRES
2026-06-20 23:23:50,216 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:23:50,217 [I] Checking connection..
2026-06-20 23:23:50,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:23:50,452 [I] Checking connection passed.
2026-06-20 23:23:50,452 [I] --------- Verifying version history table ---------
2026-06-20 23:23:50,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:23:50,689 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:23:50,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:23:50,933 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:23:50,933 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:23:50,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:23:51.168 UTC [12359] WARNING: there is no transaction in progress
2026-06-20 23:23:51,174 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:23:51,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:23:51,408 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:23:51,408 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:23:51,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:23:51,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:23:51,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5kzz.c.gcp-upstream.internal','N') ;"
2026-06-20 23:23:52,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:23:52,387 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:23:52,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:23:52,630 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:23:52,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:23:52,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:23:53,149 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:23:53,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:23:54.012 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.027 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.042 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.056 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.071 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.086 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.136 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.142 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.155 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.164 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.174 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.186 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.195 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.200 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.206 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54.211 UTC [12467] WARNING: there is no transaction in progress
2026-06-20 23:23:54,756 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:23:54,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:23:54,998 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:23:54,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:23:55,243 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:23:55,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:23:55,488 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:23:55,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:23:55,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:23:55,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5kzz.c.gcp-upstream.internal';"
2026-06-20 23:23:56,177 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:23:56,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:23:56,405 [I] DB_PATCHES have already been applied
I20260620 23:23:56.411518 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskaeh70X/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:23:56.411602 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 23:23:56.425499 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:54869
Jun 20, 2026 11:23:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:23:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:23:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:23:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:23:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:23:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:23:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:23:57.560629 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.057s user 0.003s sys 0.034s
Jun 20, 2026 11:24:17 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:24:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 11:24:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:24:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:24:17 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:24:18 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:24:18.688230 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:24:18 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997858, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:24:18.838336 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:24:18.858285 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:24:19.568874 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:24:19.568958 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-kms
I20260620 23:24:19.568991 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:56695
2026-06-20 23:24:19,917 [I] DB FLAVOR :POSTGRES
2026-06-20 23:24:19,918 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:24:19,918 [I] Checking connection
2026-06-20 23:24:19,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:24:20,151 [I] connection success
2026-06-20 23:24:20,152 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:24:20,152 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:24:20,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:24:20,374 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:24:20,375 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:24:20,375 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:24:20,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:24:20,631 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:24:21.080091 388 mini_ranger_kms.cc:310] Created kms service
I20260620 23:24:21.283121 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:24:21.327001 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:24:21 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997858, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:24:21.751544 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:24:21.751847 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskaeh70X/build/asan/bin/postgresql.jar:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskaeh70X/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:24:21.751917 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 23:24:21.765857 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12647
I20260620 23:24:21.765986 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:47685
Jun 20, 2026 11:24:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:24:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:24:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:24:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:24:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:24:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 23:24:23 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997863, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:24.003036 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.165s user 0.059s sys 0.301s
I20260620 23:24:24.003275 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:24:24.003430 388 mini_ranger_kms.cc:401] 127.0.97.20:47685/kms/v1/keys
Jun 20 23:24:24 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997863, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 23:24:24 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997863, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 23:24:27 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:24:27.060025 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36899
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36899
--ranger_config_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47685/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:24:27.381789 12698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:27.382072 12698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:27.382161 12698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:27.388917 12698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:24:27.389019 12698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:27.389061 12698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:27.389108 12698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:24:27.389147 12698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:24:27.398461 12698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:47685/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36899
--ranger_config_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:27.399644 12698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:27.400913 12698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:27.411859 12703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:27.412132 12706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:27.412164 12704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:27.412981 12698 server_base.cc:1034] running on GCE node
I20260620 23:24:27.413771 12698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:27.415097 12698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:27.416311 12698 hybrid_clock.cc:648] HybridClock initialized: now 1781997867416281 us; error 49 us; skew 500 ppm
I20260620 23:24:27.416638 12698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:27 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:27.419485 12698 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:24:27.420861 12698 webserver.cc:460] Webserver started at http://127.0.97.62:41987/ using document root <none> and password file <none>
I20260620 23:24:27.421494 12698 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:27.421625 12698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:27.421904 12698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:24:27 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:24:27.479233 12698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "e1bf83bba3c04a6c9dc54999e72139ba"
format_stamp: "Formatted at 2026-06-20 23:24:27 on dist-test-slave-5kzz"
server_key: "265e163aebff95cc9e8c383505147399"
server_key_iv: "65f48625ea2487653eddd9d3bb39a2b6"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:27.480074 12698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "e1bf83bba3c04a6c9dc54999e72139ba"
format_stamp: "Formatted at 2026-06-20 23:24:27 on dist-test-slave-5kzz"
server_key: "265e163aebff95cc9e8c383505147399"
server_key_iv: "65f48625ea2487653eddd9d3bb39a2b6"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:27.486164 12698 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 23:24:27.507870 12713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:27.509410 12698 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.007s
I20260620 23:24:27.509562 12698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "e1bf83bba3c04a6c9dc54999e72139ba"
format_stamp: "Formatted at 2026-06-20 23:24:27 on dist-test-slave-5kzz"
server_key: "265e163aebff95cc9e8c383505147399"
server_key_iv: "65f48625ea2487653eddd9d3bb39a2b6"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:27.509749 12698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:27.522925 12698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:27.526859 12698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:27.527154 12698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:27.544480 12698 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36899
I20260620 23:24:27.544502 12764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36899 every 8 connection(s)
I20260620 23:24:27.545755 12698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:24:27.549247 12765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:27.552680 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 12698
I20260620 23:24:27.552901 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:24:27.553591 388 external_mini_cluster.cc:1428] Setting key 0c743c10c1d5bfe6b4a6121f2f3e59b3
I20260620 23:24:27.561002 12765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba: Bootstrap starting.
I20260620 23:24:27.564443 12765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:27.565688 12765 log.cc:824] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:27.568243 12765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba: No bootstrap required, opened a new log
Jun 20 23:24:27 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:27.579340 12765 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } }
I20260620 23:24:27.579741 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:27.555586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.0.97.62:36899)
0620 23:24:27.556404 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:27.556418 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:27.556465 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:24:27.559325 (+ 2860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:27.559352 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:27.559806 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:27.560211 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:27.561383 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:27.563321 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:27.564471 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:27.565487 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:27.565658 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:27.569567 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:27.569575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:27.569588 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:27.569627 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:27.573844 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:27.574734 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:27.574744 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:27.574754 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:27.575094 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:27.575678 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:27.575684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:27.575687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:27.576589 (+ 902us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:27.576914 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:27.577999 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0620 23:24:27.578420 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":368,"threads_started":1}
I20260620 23:24:27.579777 12765 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:27.579928 12765 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf83bba3c04a6c9dc54999e72139ba, State: Initialized, Role: FOLLOWER
I20260620 23:24:27.580576 12765 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } }
I20260620 23:24:27.580794 12765 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:24:27.580933 12765 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:24:27.581131 12765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:27.583961 12765 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } }
I20260620 23:24:27.584448 12765 leader_election.cc:302] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1bf83bba3c04a6c9dc54999e72139ba; no voters:
I20260620 23:24:27.584982 12765 leader_election.cc:288] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:24:27.585107 12770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:24:27.585752 12770 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 1 LEADER]: Becoming Leader. State: Replica: e1bf83bba3c04a6c9dc54999e72139ba, State: Running, Role: LEADER
I20260620 23:24:27.586257 12770 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } }
I20260620 23:24:27.586969 12765 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:24:27.589500 12772 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1bf83bba3c04a6c9dc54999e72139ba. Latest consensus state: current_term: 1 leader_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } } }
I20260620 23:24:27.589898 12772 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:27.589496 12771 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } } }
I20260620 23:24:27.590659 12771 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:27.591446 12776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:24:27.596755 12776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:24:27.599560 12765 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:24:27.615324 12776 catalog_manager.cc:1347] Generated new cluster ID: 24e666d3899443e8a3f39016878765dd
I20260620 23:24:27.615466 12776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:24:27.631369 12776 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:24:27.633177 12776 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:24:27.652448 12776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba: Generated new TSK 0
I20260620 23:24:27.653555 12776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:24:28.360512 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36899
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:24:28 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 23:24:28.724063 12810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:28.724452 12810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:28.724653 12810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:28.732188 12810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:28.732363 12810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:28.732561 12810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:24:28.744231 12810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:47685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:28.745674 12810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:28.747097 12810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:28.764533 12820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:28.765125 12821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:28.770879 12810 server_base.cc:1034] running on GCE node
W20260620 23:24:28.771175 12823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:28.771893 12810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:28.773952 12810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:28.775194 12810 hybrid_clock.cc:648] HybridClock initialized: now 1781997868775149 us; error 42 us; skew 500 ppm
I20260620 23:24:28.775593 12810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:28 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:28.778968 12810 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:24:28.780570 12810 webserver.cc:460] Webserver started at http://127.0.97.1:39535/ using document root <none> and password file <none>
I20260620 23:24:28.781287 12810 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:28.781438 12810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:28.781795 12810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:24:28 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:24:28.800001 12810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "e50c1a955762468a85922df1c0ece60c"
format_stamp: "Formatted at 2026-06-20 23:24:28 on dist-test-slave-5kzz"
server_key: "10d5d2c36c16a5cc6359d7236b55a614"
server_key_iv: "473e4dcafe8c73d76c8839b5fadef606"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:28.801051 12810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e50c1a955762468a85922df1c0ece60c"
format_stamp: "Formatted at 2026-06-20 23:24:28 on dist-test-slave-5kzz"
server_key: "10d5d2c36c16a5cc6359d7236b55a614"
server_key_iv: "473e4dcafe8c73d76c8839b5fadef606"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:28.808001 12810 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 23:24:28.848634 12830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:28.851104 12810 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.004s sys 0.020s
I20260620 23:24:28.851320 12810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "e50c1a955762468a85922df1c0ece60c"
format_stamp: "Formatted at 2026-06-20 23:24:28 on dist-test-slave-5kzz"
server_key: "10d5d2c36c16a5cc6359d7236b55a614"
server_key_iv: "473e4dcafe8c73d76c8839b5fadef606"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:28.851586 12810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:28.901173 12810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:28.906318 12810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:28.906641 12810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:28.909749 12810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:28.911577 12810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:28.911716 12810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 23:24:28.911866 12810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:28.911950 12810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:29.000473 12810 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45293
I20260620 23:24:29.002140 12810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:24:29.002259 12942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45293 every 8 connection(s)
I20260620 23:24:29.011731 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 12810
I20260620 23:24:29.011955 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:24:29.012732 388 external_mini_cluster.cc:1428] Setting key 3afff8e9463c8fe64973fd09417f8c3e
Jun 20 23:24:29 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:24:29.062949 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:29.006199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36899 (local address 127.0.97.1:34377)
0620 23:24:29.017497 (+ 11298us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:29.017557 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:29.021288 (+ 3731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:29.036270 (+ 14982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:29.036286 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:29.037034 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:29.037998 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:29.038035 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:29.040083 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:29.040089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:29.042437 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:29.042453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:29.045309 (+ 2856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:29.045318 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:29.045616 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:29.046430 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:29.046460 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:29.049252 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:29.057830 (+ 8578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:29.057839 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:29.057859 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:29.058469 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:29.058961 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:29.058966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:29.058969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:29.059202 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:29.060147 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:29.060155 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:29.060610 (+ 455us) client_negotiation.cc:769] Sending connection context
0620 23:24:29.060926 (+ 316us) client_negotiation.cc:240] Negotiation successful
0620 23:24:29.061489 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10883,"thread_start_us":11782,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:24:29.065009 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:29.005453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34377 (local address 127.0.97.62:36899)
0620 23:24:29.035528 (+ 30075us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:29.035543 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:29.035569 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:24:29.035635 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:29.035640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:29.035741 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:29.038492 (+ 2751us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:29.038665 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:29.039892 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:29.043733 (+ 3841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:29.044393 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:29.046028 (+ 1635us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:29.050275 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:29.050282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:29.050287 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:29.050324 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:29.056356 (+ 6032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:29.058638 (+ 2282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:29.058643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:29.058646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:29.058827 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:29.059370 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:29.059374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:29.059378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:29.060001 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:29.064285 (+ 4284us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:29.064424 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 23:24:29.064649 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14894,"thread_start_us":15243,"threads_started":1}
I20260620 23:24:29.066720 12943 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36899
I20260620 23:24:29.067037 12943 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:29.067844 12943 heartbeater.cc:511] Master 127.0.97.62:36899 requested a full tablet report, sending...
I20260620 23:24:29.070348 12730 ts_manager.cc:194] Registered new tserver with Master: e50c1a955762468a85922df1c0ece60c (127.0.97.1:45293)
I20260620 23:24:29.072525 12730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:24:29.097862 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36899
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:24:29 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 23:24:29 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 23:24:29.540038 12949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:29.540532 12949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:29.540716 12949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:29.552223 12949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:29.552403 12949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:29.552606 12949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:24:29.566423 12949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:47685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:29.568346 12949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:29.570065 12949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:29.591986 12956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:29.593595 12957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:29.599079 12949 server_base.cc:1034] running on GCE node
W20260620 23:24:29.599752 12959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:29.600596 12949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:29.603554 12949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:29.604859 12949 hybrid_clock.cc:648] HybridClock initialized: now 1781997869604762 us; error 89 us; skew 500 ppm
I20260620 23:24:29.605269 12949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:29 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:29.609340 12949 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:24:29.623138 12949 webserver.cc:460] Webserver started at http://127.0.97.2:43773/ using document root <none> and password file <none>
I20260620 23:24:29.624054 12949 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:29.624183 12949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:29.624550 12949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:24:29 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:24:29.650307 12949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "df66dec349474b6ca25d285f36b955c0"
format_stamp: "Formatted at 2026-06-20 23:24:29 on dist-test-slave-5kzz"
server_key: "eb911324c32ada948abce93a8f1d7d13"
server_key_iv: "5e7788a5b3bfca13542c553231a03173"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:29.651589 12949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "df66dec349474b6ca25d285f36b955c0"
format_stamp: "Formatted at 2026-06-20 23:24:29 on dist-test-slave-5kzz"
server_key: "eb911324c32ada948abce93a8f1d7d13"
server_key_iv: "5e7788a5b3bfca13542c553231a03173"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:29.672818 12949 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.007s sys 0.011s
I20260620 23:24:29.707628 12970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:29.709705 12949 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.002s
I20260620 23:24:29.709903 12949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "df66dec349474b6ca25d285f36b955c0"
format_stamp: "Formatted at 2026-06-20 23:24:29 on dist-test-slave-5kzz"
server_key: "eb911324c32ada948abce93a8f1d7d13"
server_key_iv: "5e7788a5b3bfca13542c553231a03173"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:29.710136 12949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:29.733755 12949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:29.737831 12949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:29.738142 12949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:29.739214 12949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:29.740674 12949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:29.740742 12949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:29.740871 12949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:29.740922 12949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:29.780905 12949 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34453
I20260620 23:24:29.780925 13085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34453 every 8 connection(s)
I20260620 23:24:29.782212 12949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:24:29.786628 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 12949
I20260620 23:24:29.786840 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:24:29.787336 388 external_mini_cluster.cc:1428] Setting key c1bb390ee900f0bea096c310a5375739
Jun 20 23:24:29 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:29.806084 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:29.785207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51677 (local address 127.0.97.62:36899)
0620 23:24:29.785661 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:29.785665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:29.786576 (+ 911us) server_negotiation.cc:407] Connection header received
0620 23:24:29.788704 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:29.788710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:29.788817 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:29.788938 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:29.790817 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:29.791581 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:29.794123 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:29.794551 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:29.794660 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:29.798734 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:29.798744 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:29.798753 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:29.798800 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:29.802584 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:29.803351 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:29.803361 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:29.803368 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:29.803531 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:29.803942 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:29.803950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:29.803956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:29.804413 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:29.804580 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:29.805252 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 23:24:29.805540 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":250,"threads_started":1}
I20260620 23:24:29.806615 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:29.785754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36899 (local address 127.0.97.2:51677)
0620 23:24:29.786417 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:29.786459 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:29.788453 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:29.789162 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:29.789170 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:29.789730 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:29.790612 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:29.790647 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:29.791751 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:29.791759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:29.793951 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:29.793977 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:29.794796 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:29.794805 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:29.795122 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:29.795942 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:29.795976 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:29.798486 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:29.802773 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:29.802779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:29.802792 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:29.803210 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:29.803644 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:29.803647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:29.803649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:29.803813 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:29.804541 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:29.804547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:29.804841 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 23:24:29.805059 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 23:24:29.805433 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":273,"threads_started":1}
I20260620 23:24:29.808362 13086 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36899
I20260620 23:24:29.808665 13086 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:29.809334 13086 heartbeater.cc:511] Master 127.0.97.62:36899 requested a full tablet report, sending...
I20260620 23:24:29.810945 12730 ts_manager.cc:194] Registered new tserver with Master: df66dec349474b6ca25d285f36b955c0 (127.0.97.2:34453)
I20260620 23:24:29.812247 12730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51677
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:24:29.840036 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36899
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:24:30.076251 12943 heartbeater.cc:503] Master 127.0.97.62:36899 was elected leader, sending a full tablet report...
W20260620 23:24:30.156876 13092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:30.157133 13092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:30.157337 13092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:30.164394 13092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:30.164491 13092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:30.164597 13092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:24:30.173754 13092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:47685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:30.174978 13092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:30.176167 13092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:30.188970 13097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:30.188938 13098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:30.189002 13100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:30.189740 13092 server_base.cc:1034] running on GCE node
I20260620 23:24:30.190335 13092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:30.191255 13092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:30.192476 13092 hybrid_clock.cc:648] HybridClock initialized: now 1781997870192421 us; error 43 us; skew 500 ppm
I20260620 23:24:30.192780 13092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:30 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:30.196141 13092 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:24:30.197474 13092 webserver.cc:460] Webserver started at http://127.0.97.3:37675/ using document root <none> and password file <none>
I20260620 23:24:30.198122 13092 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:30.198217 13092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:30.198452 13092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:24:30 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:24:30.212296 13092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "ad805516cf9640f6aa677624d8d01419"
format_stamp: "Formatted at 2026-06-20 23:24:30 on dist-test-slave-5kzz"
server_key: "65edbd6b487e8677ce6467e26f3e58db"
server_key_iv: "32af4be33a03c9b575225e5dde6405f8"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:30.213075 13092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ad805516cf9640f6aa677624d8d01419"
format_stamp: "Formatted at 2026-06-20 23:24:30 on dist-test-slave-5kzz"
server_key: "65edbd6b487e8677ce6467e26f3e58db"
server_key_iv: "32af4be33a03c9b575225e5dde6405f8"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:30.218760 13092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:24:30.235919 13107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:30.237438 13092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:24:30.237627 13092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "ad805516cf9640f6aa677624d8d01419"
format_stamp: "Formatted at 2026-06-20 23:24:30 on dist-test-slave-5kzz"
server_key: "65edbd6b487e8677ce6467e26f3e58db"
server_key_iv: "32af4be33a03c9b575225e5dde6405f8"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:30.237879 13092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:30.260596 13092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:30.264120 13092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:30.264415 13092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:30.265336 13092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:30.266654 13092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:30.266815 13092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:30.266999 13092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:30.267071 13092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:30.299285 13092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34935
I20260620 23:24:30.299326 13219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34935 every 8 connection(s)
I20260620 23:24:30.300550 13092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:24:30.310916 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 13092
I20260620 23:24:30.311137 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:24:30.311802 388 external_mini_cluster.cc:1428] Setting key 4fc797416254ac5de44e4dc8451472f1
Jun 20 23:24:30 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:30.319630 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:30.303075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46151 (local address 127.0.97.62:36899)
0620 23:24:30.303248 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:30.303252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:30.304373 (+ 1121us) server_negotiation.cc:407] Connection header received
0620 23:24:30.306114 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:30.306119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:30.306219 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:30.306336 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:30.307749 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:30.308526 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:30.310004 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:30.310386 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:30.310598 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:30.314249 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:30.314258 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:30.314265 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:30.314325 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:30.317150 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:30.317772 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:30.317776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:30.317778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:30.317900 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:30.318261 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:30.318263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:30.318265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:30.318507 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:30.318612 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:30.319233 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 23:24:30.319406 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:24:30.320502 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:30.303533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36899 (local address 127.0.97.3:46151)
0620 23:24:30.304217 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:30.304259 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:30.305850 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:30.306561 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:30.306569 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:30.307045 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:30.307587 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:30.307604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:30.308679 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:30.308683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:30.309865 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:30.309876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:30.310737 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:30.310745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:30.311070 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:30.311774 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:30.311798 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:30.314045 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:30.317269 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:30.317275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:30.317286 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:30.317679 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:30.317988 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:30.317991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:30.317993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:30.318172 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:30.318606 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:30.318612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:30.318943 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 23:24:30.319159 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 23:24:30.319497 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":277,"threads_started":1}
I20260620 23:24:30.321872 13220 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36899
I20260620 23:24:30.322144 13220 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:30.322852 13220 heartbeater.cc:511] Master 127.0.97.62:36899 requested a full tablet report, sending...
I20260620 23:24:30.324137 12730 ts_manager.cc:194] Registered new tserver with Master: ad805516cf9640f6aa677624d8d01419 (127.0.97.3:34935)
I20260620 23:24:30.324878 12730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46151
I20260620 23:24:30.327127 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:24:30.328951 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 12810
Jun 20 23:24:30 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:24:30.344337 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:30.329726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.0.97.1:45293)
0620 23:24:30.330226 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:30.330231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:30.330255 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:24:30.330322 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:30.330328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:30.330507 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:30.330624 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:30.332026 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:30.333008 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:30.334385 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:30.334961 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:30.335058 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:30.339026 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:30.339042 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:30.339059 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:30.339098 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:30.341817 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:30.342461 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:30.342467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:30.342469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:30.342596 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:30.342996 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:30.343000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:30.343002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:30.343283 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:30.343503 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:30.343750 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 23:24:30.343925 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":239,"threads_started":1}
I20260620 23:24:30.345507 12938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:30.427248 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 12810
I20260620 23:24:30.443185 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 12949
Jun 20 23:24:30 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:24:30.459632 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:30.444010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.0.97.2:34453)
0620 23:24:30.444531 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:30.444537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:30.444560 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:24:30.444618 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:30.444624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:30.444835 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:30.445035 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:30.446313 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:30.447395 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:30.448957 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:30.449551 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:30.449697 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:30.454119 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:30.454129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:30.454143 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:30.454180 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:30.456864 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:30.457575 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:30.457582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:30.457584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:30.457718 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:30.458090 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:30.458097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:30.458100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:30.458430 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:30.458879 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:30.459035 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 23:24:30.459293 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":253,"threads_started":1}
I20260620 23:24:30.460850 13081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:30.544848 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 12949
I20260620 23:24:30.565392 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 13092
Jun 20 23:24:30 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:24:30.581722 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:30.566399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.0.97.3:34935)
0620 23:24:30.567110 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:30.567116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:30.567145 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:24:30.567201 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:30.567207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:30.567419 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:30.567589 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:30.568889 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:30.569888 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:30.571356 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:30.571908 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:30.572047 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:30.576222 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:30.576234 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:30.576249 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:30.576282 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:30.578950 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:30.579642 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:30.579650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:30.579652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:30.579774 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:30.580154 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:30.580160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:30.580163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:30.580487 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:30.580853 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:30.581124 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 23:24:30.581385 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":420,"threads_started":1}
I20260620 23:24:30.583002 13215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:30.664615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 13092
I20260620 23:24:30.680112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 12698
I20260620 23:24:30.681358 12760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:30.785578 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 12698
I20260620 23:24:30.808936 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36899
--webserver_interface=127.0.97.62
--webserver_port=41987
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36899
--ranger_config_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47685/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47685/kms with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:24:31.121114 13231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:31.121400 13231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:31.121495 13231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:31.128479 13231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:24:31.128579 13231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:31.128621 13231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:31.128666 13231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:24:31.128705 13231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:24:31.138422 13231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47685/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36899
--ranger_config_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:31.139672 13231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:31.140911 13231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:31.152277 13237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:31.152370 13236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:31.152346 13239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:31.153349 13231 server_base.cc:1034] running on GCE node
I20260620 23:24:31.154147 13231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:31.155597 13231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:31.156800 13231 hybrid_clock.cc:648] HybridClock initialized: now 1781997871156761 us; error 29 us; skew 500 ppm
I20260620 23:24:31.157096 13231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:31 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:31.160045 13231 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:24:31.161455 13231 webserver.cc:460] Webserver started at http://127.0.97.62:41987/ using document root <none> and password file <none>
I20260620 23:24:31.162034 13231 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:31.162133 13231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:24:31 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:24:31.253793 13231 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:24:31.257570 13247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:31.259279 13231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:24:31.259470 13231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "e1bf83bba3c04a6c9dc54999e72139ba"
format_stamp: "Formatted at 2026-06-20 23:24:27 on dist-test-slave-5kzz"
server_key: "265e163aebff95cc9e8c383505147399"
server_key_iv: "65f48625ea2487653eddd9d3bb39a2b6"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:31.260829 13231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:31.280668 13231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:31.284797 13231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:31.285079 13231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:31.304339 13231 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36899
I20260620 23:24:31.304359 13298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36899 every 8 connection(s)
I20260620 23:24:31.305797 13231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:24:31.310925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 13231
I20260620 23:24:31.312084 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45293
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:36899
--webserver_port=39535
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47685/kms with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:24:31.312605 13299 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:24:31.315775 13299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba: Bootstrap starting.
I20260620 23:24:31.340526 13299 log.cc:824] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:31.351605 13299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:24:31.352228 13299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba: Bootstrap complete.
I20260620 23:24:31.364856 13299 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } }
I20260620 23:24:31.366055 13299 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf83bba3c04a6c9dc54999e72139ba, State: Initialized, Role: FOLLOWER
I20260620 23:24:31.366897 13299 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } }
I20260620 23:24:31.367166 13299 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:24:31.367296 13299 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:24:31.367476 13299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:24:31.373013 13299 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } }
I20260620 23:24:31.373559 13299 leader_election.cc:302] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1bf83bba3c04a6c9dc54999e72139ba; no voters:
I20260620 23:24:31.374200 13299 leader_election.cc:288] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:24:31.374413 13303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:24:31.382565 13299 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:24:31.382959 13303 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [term 2 LEADER]: Becoming Leader. State: Replica: e1bf83bba3c04a6c9dc54999e72139ba, State: Running, Role: LEADER
I20260620 23:24:31.383651 13303 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } }
I20260620 23:24:31.393455 13305 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1bf83bba3c04a6c9dc54999e72139ba. Latest consensus state: current_term: 2 leader_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } } }
I20260620 23:24:31.393512 13304 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bf83bba3c04a6c9dc54999e72139ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36899 } } }
I20260620 23:24:31.393857 13305 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:31.393854 13304 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:31.413457 13308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:24:31.426858 13308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:24:31.428310 13308 catalog_manager.cc:1259] Loaded cluster ID: 24e666d3899443e8a3f39016878765dd
I20260620 23:24:31.428459 13308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:24:31.432608 13308 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:24:31.434494 13308 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba: Loaded TSK: 0
I20260620 23:24:31.436388 13308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:24:31.483901 13299 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 23:24:31.760025 13301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:31.760512 13301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:31.760733 13301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:31.772415 13301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:31.772624 13301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:31.772872 13301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:24:31.788206 13301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39535
--tserver_master_addrs=127.0.97.62:36899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:31.790035 13301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:31.792068 13301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:31.845583 13336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:31.845614 13337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:31.851001 13301 server_base.cc:1034] running on GCE node
W20260620 23:24:31.852356 13339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:31.853282 13301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:31.858593 13301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:31.859958 13301 hybrid_clock.cc:648] HybridClock initialized: now 1781997871859876 us; error 74 us; skew 500 ppm
I20260620 23:24:31.860363 13301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:31 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:31.864995 13301 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:24:31.886650 13301 webserver.cc:460] Webserver started at http://127.0.97.1:39535/ using document root <none> and password file <none>
I20260620 23:24:31.887763 13301 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:31.887975 13301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:24:31 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:24:31.937593 13301 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.011s sys 0.003s
I20260620 23:24:31.941908 13347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:31.943928 13301 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:24:31.944175 13301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "e50c1a955762468a85922df1c0ece60c"
format_stamp: "Formatted at 2026-06-20 23:24:28 on dist-test-slave-5kzz"
server_key: "10d5d2c36c16a5cc6359d7236b55a614"
server_key_iv: "473e4dcafe8c73d76c8839b5fadef606"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:31.945619 13301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:31.988722 13301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:31.994081 13301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:31.994451 13301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:31.996284 13301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:32.004369 13301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:32.004487 13301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:32.004698 13301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:32.004851 13301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:32.095785 13301 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45293
I20260620 23:24:32.095832 13459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45293 every 8 connection(s)
I20260620 23:24:32.097391 13301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:24:32.099321 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 13301
I20260620 23:24:32.100124 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34453
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:36899
--webserver_port=43773
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47685/kms with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:24:32 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:32.146054 13464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:32.110320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36899 (local address 127.0.97.1:47941)
0620 23:24:32.116296 (+ 5976us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:32.116369 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:32.118857 (+ 2488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:32.121670 (+ 2813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:32.121684 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:32.122404 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:32.123353 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:32.123386 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:32.126229 (+ 2843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:32.126237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:32.128745 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:32.128766 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:32.130374 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:32.130385 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:32.130794 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:32.131718 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:32.131788 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:32.135152 (+ 3364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:32.140566 (+ 5414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:32.140577 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:32.140595 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:32.141263 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:32.141862 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:32.141868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:32.141872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:32.142134 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:32.143226 (+ 1092us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:32.143237 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:32.143666 (+ 429us) client_negotiation.cc:769] Sending connection context
0620 23:24:32.143972 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 23:24:32.144592 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5625,"thread_start_us":5373,"threads_started":1}
I20260620 23:24:32.149562 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:32.111531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47941 (local address 127.0.97.62:36899)
0620 23:24:32.119813 (+ 8282us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:32.119835 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:32.119890 (+ 55us) server_negotiation.cc:407] Connection header received
0620 23:24:32.120388 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:32.120410 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:32.121002 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:32.121460 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:32.123661 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:32.125986 (+ 2325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:32.128943 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:32.130090 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:32.130308 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:32.135549 (+ 5241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:32.135562 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:32.135590 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:32.135651 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:32.140369 (+ 4718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:32.141484 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:32.141494 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:32.141503 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:32.141714 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:32.142314 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:32.142320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:32.142324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:32.143036 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:32.147416 (+ 4380us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:32.147705 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 23:24:32.148156 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7960,"thread_start_us":7712,"threads_started":1}
I20260620 23:24:32.153649 13460 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36899
I20260620 23:24:32.153983 13460 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:32.154911 13460 heartbeater.cc:511] Master 127.0.97.62:36899 requested a full tablet report, sending...
W20260620 23:24:32.158268 13460 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36899 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:24:32.158510 13460 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:32.158847 13460 heartbeater.cc:511] Master 127.0.97.62:36899 requested a full tablet report, sending...
I20260620 23:24:32.160177 13460 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:32.160446 13460 heartbeater.cc:511] Master 127.0.97.62:36899 requested a full tablet report, sending...
W20260620 23:24:32.161621 13460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:24:32.548802 13463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:32.549201 13463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:32.549387 13463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:32.561255 13463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:32.561431 13463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:32.561645 13463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:24:32.577663 13463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43773
--tserver_master_addrs=127.0.97.62:36899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:32.579737 13463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:32.581544 13463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:32.605406 13478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:32.605432 13480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:32.606146 13477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:32.606769 13463 server_base.cc:1034] running on GCE node
I20260620 23:24:32.607538 13463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:32.608778 13463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:32.610038 13463 hybrid_clock.cc:648] HybridClock initialized: now 1781997872609960 us; error 60 us; skew 500 ppm
I20260620 23:24:32.610446 13463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:32 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:32.614405 13463 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:24:32.616122 13463 webserver.cc:460] Webserver started at http://127.0.97.2:43773/ using document root <none> and password file <none>
I20260620 23:24:32.616842 13463 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:32.616995 13463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:24:32 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:24:32.643343 13463 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:24:32.646589 13490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:32.648249 13463 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:24:32.648445 13463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "df66dec349474b6ca25d285f36b955c0"
format_stamp: "Formatted at 2026-06-20 23:24:29 on dist-test-slave-5kzz"
server_key: "eb911324c32ada948abce93a8f1d7d13"
server_key_iv: "5e7788a5b3bfca13542c553231a03173"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:32.649493 13463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:32.671445 13463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:32.675714 13463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:32.676016 13463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:32.677089 13463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:32.678586 13463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:32.678732 13463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:32.678859 13463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:32.678946 13463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:32.789979 13463 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34453
I20260620 23:24:32.790041 13604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34453 every 8 connection(s)
I20260620 23:24:32.792025 13463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:24:32.796476 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 13463
I20260620 23:24:32.797245 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34935
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:36899
--webserver_port=37675
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47685/kms with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:24:32 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:24:32 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:32.828814 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:32.795548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36899 (local address 127.0.97.2:36279)
0620 23:24:32.803305 (+ 7757us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:32.803373 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:32.806138 (+ 2765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:32.807524 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:32.807536 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:32.808130 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:32.808818 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:32.808857 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:32.810396 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:32.810404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:32.812705 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:32.812721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:32.814165 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:32.814174 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:32.814501 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:32.815407 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:32.815447 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:32.819437 (+ 3990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:32.823672 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:32.823681 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:32.823701 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:32.824339 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:32.824793 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:32.824800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:32.824806 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:32.825093 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:32.825912 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:32.825920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:32.826442 (+ 522us) client_negotiation.cc:769] Sending connection context
0620 23:24:32.826835 (+ 393us) client_negotiation.cc:240] Negotiation successful
0620 23:24:32.827406 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7386,"thread_start_us":7829,"threads_started":1}
I20260620 23:24:32.829888 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:32.794881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36279 (local address 127.0.97.62:36899)
0620 23:24:32.806946 (+ 12065us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:32.806952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:32.806973 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:24:32.807038 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:32.807044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:32.807132 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:32.807249 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:32.809028 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:32.810239 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:32.812899 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:32.813683 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:32.813835 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:32.819756 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:32.819763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:32.819768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:32.819819 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:32.823478 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:32.824490 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:32.824496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:32.824500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:32.824653 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:32.825227 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:32.825232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:32.825235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:32.825675 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:32.825833 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:32.829294 (+ 3461us) server_negotiation.cc:299] Negotiation successful
0620 23:24:32.829541 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11926,"thread_start_us":11798,"threads_started":1}
I20260620 23:24:32.832204 13605 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36899
I20260620 23:24:32.832547 13605 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:32.833386 13605 heartbeater.cc:511] Master 127.0.97.62:36899 requested a full tablet report, sending...
I20260620 23:24:32.835464 13264 ts_manager.cc:194] Registered new tserver with Master: df66dec349474b6ca25d285f36b955c0 (127.0.97.2:34453)
I20260620 23:24:32.837586 13264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36279
I20260620 23:24:33.161841 13460 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:33.162649 13460 heartbeater.cc:511] Master 127.0.97.62:36899 requested a full tablet report, sending...
I20260620 23:24:33.164778 13264 ts_manager.cc:194] Registered new tserver with Master: e50c1a955762468a85922df1c0ece60c (127.0.97.1:45293)
I20260620 23:24:33.166482 13264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47941
Jun 20 23:24:33 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 23:24:33.285422 13609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:33.285936 13609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:33.286118 13609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:33.297497 13609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:33.297739 13609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:33.297956 13609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:24:33.313630 13609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:47685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37675
--tserver_master_addrs=127.0.97.62:36899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:33.315586 13609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:33.317255 13609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:33.353626 13616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:33.363215 13615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:33.367460 13619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:33.368184 13609 server_base.cc:1034] running on GCE node
I20260620 23:24:33.369081 13609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:33.389238 13609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:33.392843 13609 hybrid_clock.cc:648] HybridClock initialized: now 1781997873392112 us; error 715 us; skew 500 ppm
I20260620 23:24:33.393280 13609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:33 dist-test-slave-5kzz krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:33.397642 13609 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:24:33.439210 13609 webserver.cc:460] Webserver started at http://127.0.97.3:37675/ using document root <none> and password file <none>
I20260620 23:24:33.440490 13609 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:33.440735 13609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:24:33 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:24:33.492432 13609 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.006s sys 0.009s
I20260620 23:24:33.502950 13626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:33.504925 13609 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.008s sys 0.002s
I20260620 23:24:33.505963 13609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "ad805516cf9640f6aa677624d8d01419"
format_stamp: "Formatted at 2026-06-20 23:24:30 on dist-test-slave-5kzz"
server_key: "65edbd6b487e8677ce6467e26f3e58db"
server_key_iv: "32af4be33a03c9b575225e5dde6405f8"
server_key_version: "kuduclusterkey@0"
I20260620 23:24:33.509572 13609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:33.579851 13609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:33.585566 13609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:33.585898 13609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:33.608196 13609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Jun 20 23:24:33 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:24:33.616420 13609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:33.616582 13609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 23:24:33.616732 13609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:33.616888 13609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:33.842381 13605 heartbeater.cc:503] Master 127.0.97.62:36899 was elected leader, sending a full tablet report...
I20260620 23:24:33.971401 13609 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34935
I20260620 23:24:33.971490 13746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34935 every 8 connection(s)
I20260620 23:24:33.972831 13609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:24:33.974272 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 13609
Jun 20 23:24:33 dist-test-slave-5kzz krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:34.005844 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:33.975247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.0.97.62:36899)
0620 23:24:33.986300 (+ 11053us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:33.986311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:33.989892 (+ 3581us) server_negotiation.cc:407] Connection header received
0620 23:24:33.990075 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:33.990078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:33.990126 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:33.990225 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:33.992159 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:33.993181 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:33.995100 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:33.995980 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:33.996163 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:33.999420 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:33.999425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:33.999428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:33.999456 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:34.003178 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:34.003853 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:34.003856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:34.003858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:34.003978 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:34.004416 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:34.004422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:34.004425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:34.004786 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:34.005003 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:34.005358 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 23:24:34.005573 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":10863,"spinlock_wait_cycles":5888,"thread_start_us":11301,"threads_started":1}
I20260620 23:24:34.006881 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:33.975486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38849 (local address 127.0.97.62:36899)
0620 23:24:33.986851 (+ 11365us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:33.986859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:33.987470 (+ 611us) server_negotiation.cc:407] Connection header received
0620 23:24:33.989400 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:33.989410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:33.989520 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:33.989650 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:33.991710 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:33.992794 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:33.995084 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:33.995985 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:33.996179 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.000065 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:34.000070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:34.000074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:34.000102 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:34.003583 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:34.004416 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:34.004422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:34.004425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:34.004572 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:34.005010 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:34.005017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:34.005023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:34.005357 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:34.005491 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:34.006316 (+ 825us) server_negotiation.cc:299] Negotiation successful
0620 23:24:34.006478 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":210,"server-negotiator.queue_time_us":11221,"spinlock_wait_cycles":5760,"thread_start_us":10827,"threads_started":1}
I20260620 23:24:34.007967 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:33.976283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36899 (local address 127.0.97.3:38849)
0620 23:24:33.987264 (+ 10981us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:33.987326 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:33.989106 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:33.989942 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:33.989955 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:33.990629 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:33.991536 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:33.991564 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:33.992949 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:33.992952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:33.994871 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:33.994887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:33.996228 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:33.996236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:33.996546 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:33.997246 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:33.997309 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:33.999754 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:34.003719 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:34.003727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:34.003743 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:34.004290 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:34.004673 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:34.004676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:34.004678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:34.004887 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:34.005496 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:34.005504 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:34.005908 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 23:24:34.006219 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 23:24:34.006649 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10692,"thread_start_us":10454,"threads_started":1}
I20260620 23:24:34.009228 13747 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36899
I20260620 23:24:34.009469 13747 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:34.010072 13747 heartbeater.cc:511] Master 127.0.97.62:36899 requested a full tablet report, sending...
I20260620 23:24:34.011317 13264 ts_manager.cc:194] Registered new tserver with Master: ad805516cf9640f6aa677624d8d01419 (127.0.97.3:34935)
I20260620 23:24:34.012072 13264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38849
I20260620 23:24:34.020509 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:24:34.035867 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.023449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.0.97.62:36899)
0620 23:24:34.023680 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:34.023688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:34.023949 (+ 261us) server_negotiation.cc:407] Connection header received
0620 23:24:34.024138 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:34.024149 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:34.024214 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:34.024316 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:34.025867 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.026608 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.028020 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.028439 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.028575 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.030997 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:34.031002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:34.031005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:34.031042 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:34.033314 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:34.033960 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:34.033966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:34.033970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:34.034098 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:34.034473 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:34.034479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:34.034483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:34.034791 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:34.035051 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:34.035345 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 23:24:34.035565 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 23:24:34.040077 13264 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:24:34.047349 13264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:24:34.071336 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.060342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.0.97.2:34453)
0620 23:24:34.063147 (+ 2805us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:34.063156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:34.063184 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:24:34.063299 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:34.063306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:34.063542 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:34.064518 (+ 976us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:34.064794 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.066005 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.069267 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.070175 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.070328 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.070555 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:34.070836 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:24:34.071009 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2645,"thread_start_us":2486,"threads_started":1}
I20260620 23:24:34.071488 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.060207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34453 (local address 127.0.0.1:33456)
0620 23:24:34.061157 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:34.061213 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:34.061422 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:34.063782 (+ 2360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:34.063790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:34.063823 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:34.064308 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:34.064327 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.066169 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.066177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:34.069136 (+ 2959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:34.069145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.070379 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.070385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:34.070563 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.070619 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 23:24:34.070796 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 23:24:34.070994 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":109,"thread_start_us":423,"threads_started":1}
I20260620 23:24:34.072343 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.060994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.0.97.3:34935)
0620 23:24:34.062810 (+ 1816us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:34.062822 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:34.062854 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:24:34.062958 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:34.062965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:34.063205 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:34.063413 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:34.064481 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.066062 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.069067 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.070458 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.070592 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.070828 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:34.071812 (+ 984us) server_negotiation.cc:299] Negotiation successful
0620 23:24:34.072011 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1566,"thread_start_us":2027,"threads_started":1}
I20260620 23:24:34.072731 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.062621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45293 (local address 127.0.0.1:43166)
0620 23:24:34.063527 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:34.063546 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:34.063644 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:34.064786 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:34.064790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:34.064815 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:34.065186 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:34.065196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.066989 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.066996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:34.068852 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:34.068864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.070728 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.070735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:34.070927 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.070972 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 23:24:34.071042 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 23:24:34.071169 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":50,"thread_start_us":658,"threads_started":1}
I20260620 23:24:34.072733 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.063097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.0.97.1:45293)
0620 23:24:34.064233 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:34.064242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:34.064268 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:24:34.064356 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:34.064364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:34.064586 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:34.065376 (+ 790us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:34.065611 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.066865 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.069007 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.070458 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.071981 (+ 1523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.072197 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:34.072261 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 23:24:34.072433 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":753,"threads_started":1}
I20260620 23:24:34.072968 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.060344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34935 (local address 127.0.0.1:50506)
0620 23:24:34.062493 (+ 2149us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:34.062511 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:34.062609 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:34.063728 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:34.063737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:34.063767 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:34.064275 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:34.064292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.066225 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.066229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:34.068919 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:34.068932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.071011 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.071019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:34.071176 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.071204 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:24:34.071285 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:24:34.071419 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2033,"mutex_wait_us":99,"thread_start_us":1924,"threads_started":1}
I20260620 23:24:34.075343 13540 tablet_service.cc:1467] Processing CreateTablet for tablet c8efed6465ac4726af6e44c253dae532 (DEFAULT_TABLE table=test-table [id=a745bf985d624fe0b3d28cf4e99012ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:24:34.076057 13677 tablet_service.cc:1467] Processing CreateTablet for tablet c8efed6465ac4726af6e44c253dae532 (DEFAULT_TABLE table=test-table [id=a745bf985d624fe0b3d28cf4e99012ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:24:34.076804 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8efed6465ac4726af6e44c253dae532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:34.076560 13395 tablet_service.cc:1467] Processing CreateTablet for tablet c8efed6465ac4726af6e44c253dae532 (DEFAULT_TABLE table=test-table [id=a745bf985d624fe0b3d28cf4e99012ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:24:34.077340 13677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8efed6465ac4726af6e44c253dae532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:34.077832 13395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8efed6465ac4726af6e44c253dae532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:34.088797 13768 tablet_bootstrap.cc:492] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419: Bootstrap starting.
I20260620 23:24:34.092051 13769 tablet_bootstrap.cc:492] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c: Bootstrap starting.
I20260620 23:24:34.093428 13768 tablet_bootstrap.cc:654] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:34.094656 13768 log.cc:824] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:34.095723 13769 tablet_bootstrap.cc:654] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:34.096091 13770 tablet_bootstrap.cc:492] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0: Bootstrap starting.
I20260620 23:24:34.097164 13769 log.cc:824] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:34.099772 13770 tablet_bootstrap.cc:654] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:34.100443 13769 tablet_bootstrap.cc:492] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c: No bootstrap required, opened a new log
I20260620 23:24:34.100718 13769 ts_tablet_manager.cc:1397] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 23:24:34.100921 13770 log.cc:824] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:34.104406 13770 tablet_bootstrap.cc:492] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0: No bootstrap required, opened a new log
I20260620 23:24:34.104537 13768 tablet_bootstrap.cc:492] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419: No bootstrap required, opened a new log
I20260620 23:24:34.104662 13770 ts_tablet_manager.cc:1397] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:24:34.104791 13768 ts_tablet_manager.cc:1397] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20260620 23:24:34.111534 13769 raft_consensus.cc:348] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df66dec349474b6ca25d285f36b955c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34453 } } peers { permanent_uuid: "ad805516cf9640f6aa677624d8d01419" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34935 } } peers { permanent_uuid: "e50c1a955762468a85922df1c0ece60c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45293 } }
I20260620 23:24:34.111725 13770 raft_consensus.cc:348] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df66dec349474b6ca25d285f36b955c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34453 } } peers { permanent_uuid: "ad805516cf9640f6aa677624d8d01419" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34935 } } peers { permanent_uuid: "e50c1a955762468a85922df1c0ece60c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45293 } }
I20260620 23:24:34.111974 13769 raft_consensus.cc:374] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:34.112061 13769 raft_consensus.cc:729] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50c1a955762468a85922df1c0ece60c, State: Initialized, Role: FOLLOWER
I20260620 23:24:34.112124 13770 raft_consensus.cc:374] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:34.112275 13770 raft_consensus.cc:729] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df66dec349474b6ca25d285f36b955c0, State: Initialized, Role: FOLLOWER
I20260620 23:24:34.112843 13770 consensus_queue.cc:260] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df66dec349474b6ca25d285f36b955c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34453 } } peers { permanent_uuid: "ad805516cf9640f6aa677624d8d01419" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34935 } } peers { permanent_uuid: "e50c1a955762468a85922df1c0ece60c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45293 } }
I20260620 23:24:34.113106 13769 consensus_queue.cc:260] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df66dec349474b6ca25d285f36b955c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34453 } } peers { permanent_uuid: "ad805516cf9640f6aa677624d8d01419" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34935 } } peers { permanent_uuid: "e50c1a955762468a85922df1c0ece60c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45293 } }
I20260620 23:24:34.114574 13770 ts_tablet_manager.cc:1428] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0: Time spent starting tablet: real 0.010s user 0.000s sys 0.011s
I20260620 23:24:34.115677 13768 raft_consensus.cc:348] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df66dec349474b6ca25d285f36b955c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34453 } } peers { permanent_uuid: "ad805516cf9640f6aa677624d8d01419" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34935 } } peers { permanent_uuid: "e50c1a955762468a85922df1c0ece60c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45293 } }
I20260620 23:24:34.116175 13768 raft_consensus.cc:374] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:34.116297 13768 raft_consensus.cc:729] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad805516cf9640f6aa677624d8d01419, State: Initialized, Role: FOLLOWER
I20260620 23:24:34.117005 13768 consensus_queue.cc:260] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df66dec349474b6ca25d285f36b955c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34453 } } peers { permanent_uuid: "ad805516cf9640f6aa677624d8d01419" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34935 } } peers { permanent_uuid: "e50c1a955762468a85922df1c0ece60c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45293 } }
I20260620 23:24:34.118301 13747 heartbeater.cc:503] Master 127.0.97.62:36899 was elected leader, sending a full tablet report...
I20260620 23:24:34.119275 13768 ts_tablet_manager.cc:1428] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419: Time spent starting tablet: real 0.014s user 0.013s sys 0.003s
I20260620 23:24:34.120297 13460 heartbeater.cc:503] Master 127.0.97.62:36899 was elected leader, sending a full tablet report...
I20260620 23:24:34.120828 13769 ts_tablet_manager.cc:1428] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c: Time spent starting tablet: real 0.020s user 0.021s sys 0.000s
W20260620 23:24:34.224550 13748 tablet.cc:2369] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:24:34.308383 13606 tablet.cc:2369] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:24:34.340824 13776 raft_consensus.cc:482] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:24:34.341115 13776 raft_consensus.cc:504] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df66dec349474b6ca25d285f36b955c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34453 } } peers { permanent_uuid: "ad805516cf9640f6aa677624d8d01419" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34935 } } peers { permanent_uuid: "e50c1a955762468a85922df1c0ece60c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45293 } }
I20260620 23:24:34.342900 13776 leader_election.cc:288] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df66dec349474b6ca25d285f36b955c0 (127.0.97.2:34453), e50c1a955762468a85922df1c0ece60c (127.0.97.1:45293)
I20260620 23:24:34.349422 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.343256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45293 (local address 127.0.97.3:59261)
0620 23:24:34.343443 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:34.343466 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:34.343606 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:34.344084 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:34.344088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:34.344122 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:34.344377 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:34.344392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.345642 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.345646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:34.347547 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:34.347561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.348897 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.348901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:34.349018 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.349040 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:24:34.349108 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 23:24:34.349195 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 23:24:34.349558 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.343322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59261 (local address 127.0.97.1:45293)
0620 23:24:34.343564 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:34.343570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:34.343594 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:24:34.343771 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:34.343779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:34.343926 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:34.344043 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:34.344544 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.345516 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.347699 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.348744 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.348875 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.348976 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:34.349162 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 23:24:34.349290 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 23:24:34.350230 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.343275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34453 (local address 127.0.97.3:59955)
0620 23:24:34.344316 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:34.344338 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:34.344452 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:34.344794 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:34.344796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:34.344814 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:34.345069 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:34.345076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.346180 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.346185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:34.347989 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:34.347999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.349540 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.349548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:34.349705 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.349746 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:24:34.349840 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:24:34.349945 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":253,"thread_start_us":413,"threads_started":1}
I20260620 23:24:34.350419 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.343422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59955 (local address 127.0.97.2:34453)
0620 23:24:34.343620 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:34.343625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:34.344424 (+ 799us) server_negotiation.cc:407] Connection header received
0620 23:24:34.344584 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:34.344590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:34.344681 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:34.344777 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:34.345207 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.346039 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.348127 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.349334 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.349463 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.349554 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:34.350038 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 23:24:34.350168 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:24:34.350787 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8efed6465ac4726af6e44c253dae532" candidate_uuid: "ad805516cf9640f6aa677624d8d01419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50c1a955762468a85922df1c0ece60c" is_pre_election: true
I20260620 23:24:34.351373 13415 raft_consensus.cc:2454] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad805516cf9640f6aa677624d8d01419 in term 0.
I20260620 23:24:34.351522 13560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8efed6465ac4726af6e44c253dae532" candidate_uuid: "ad805516cf9640f6aa677624d8d01419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df66dec349474b6ca25d285f36b955c0" is_pre_election: true
I20260620 23:24:34.352023 13628 leader_election.cc:302] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad805516cf9640f6aa677624d8d01419, e50c1a955762468a85922df1c0ece60c; no voters:
I20260620 23:24:34.352046 13560 raft_consensus.cc:2454] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad805516cf9640f6aa677624d8d01419 in term 0.
I20260620 23:24:34.352439 13776 raft_consensus.cc:2781] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:24:34.352620 13776 raft_consensus.cc:482] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:24:34.352741 13776 raft_consensus.cc:3037] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:24:34.353730 13461 tablet.cc:2369] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:24:34.355369 13776 raft_consensus.cc:504] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df66dec349474b6ca25d285f36b955c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34453 } } peers { permanent_uuid: "ad805516cf9640f6aa677624d8d01419" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34935 } } peers { permanent_uuid: "e50c1a955762468a85922df1c0ece60c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45293 } }
I20260620 23:24:34.355916 13776 leader_election.cc:288] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [CANDIDATE]: Term 1 election: Requested vote from peers df66dec349474b6ca25d285f36b955c0 (127.0.97.2:34453), e50c1a955762468a85922df1c0ece60c (127.0.97.1:45293)
I20260620 23:24:34.356339 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8efed6465ac4726af6e44c253dae532" candidate_uuid: "ad805516cf9640f6aa677624d8d01419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50c1a955762468a85922df1c0ece60c"
I20260620 23:24:34.356345 13560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8efed6465ac4726af6e44c253dae532" candidate_uuid: "ad805516cf9640f6aa677624d8d01419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df66dec349474b6ca25d285f36b955c0"
I20260620 23:24:34.356545 13415 raft_consensus.cc:3037] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:34.356545 13560 raft_consensus.cc:3037] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:34.359655 13560 raft_consensus.cc:2454] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad805516cf9640f6aa677624d8d01419 in term 1.
I20260620 23:24:34.359655 13415 raft_consensus.cc:2454] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad805516cf9640f6aa677624d8d01419 in term 1.
I20260620 23:24:34.360059 13629 leader_election.cc:302] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad805516cf9640f6aa677624d8d01419, df66dec349474b6ca25d285f36b955c0; no voters:
I20260620 23:24:34.360349 13776 raft_consensus.cc:2781] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:24:34.360847 13776 raft_consensus.cc:686] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 1 LEADER]: Becoming Leader. State: Replica: ad805516cf9640f6aa677624d8d01419, State: Running, Role: LEADER
I20260620 23:24:34.361438 13776 consensus_queue.cc:237] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df66dec349474b6ca25d285f36b955c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34453 } } peers { permanent_uuid: "ad805516cf9640f6aa677624d8d01419" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34935 } } peers { permanent_uuid: "e50c1a955762468a85922df1c0ece60c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45293 } }
I20260620 23:24:34.365658 13262 catalog_manager.cc:5495] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad805516cf9640f6aa677624d8d01419 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ad805516cf9640f6aa677624d8d01419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df66dec349474b6ca25d285f36b955c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad805516cf9640f6aa677624d8d01419" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e50c1a955762468a85922df1c0ece60c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45293 } health_report { overall_health: UNKNOWN } } }
I20260620 23:24:34.435737 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.430660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.0.97.3:34935)
0620 23:24:34.430867 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:34.430872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:34.430890 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:24:34.430987 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:34.430991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:34.431068 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:34.431192 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:24:34.431514 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.432229 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.433528 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.433914 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.434004 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.434504 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:24:34.435129 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:24:34.435293 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:34.435391 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 23:24:34.435482 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:24:34.443809 13560 raft_consensus.cc:1261] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0 [term 1 FOLLOWER]: Refusing update from remote peer ad805516cf9640f6aa677624d8d01419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:24:34.443809 13415 raft_consensus.cc:1261] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c [term 1 FOLLOWER]: Refusing update from remote peer ad805516cf9640f6aa677624d8d01419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:24:34.444880 13778 consensus_queue.cc:1035] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df66dec349474b6ca25d285f36b955c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:24:34.445200 13776 consensus_queue.cc:1035] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e50c1a955762468a85922df1c0ece60c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:24:34.456246 13781 mvcc.cc:205] Tried to move back new op lower bound from 7299063293711773696 to 7299063293388161024. Current Snapshot: MvccSnapshot[applied={T|T < 7299063293711773696}]
I20260620 23:24:34.469849 13263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:24:34.470163 13263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:24:34.473868 13263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1bf83bba3c04a6c9dc54999e72139ba: Sending DeleteTablet for 3 replicas of tablet c8efed6465ac4726af6e44c253dae532
I20260620 23:24:34.474998 13540 tablet_service.cc:1514] Processing DeleteTablet for tablet c8efed6465ac4726af6e44c253dae532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:24:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33456
I20260620 23:24:34.475240 13395 tablet_service.cc:1514] Processing DeleteTablet for tablet c8efed6465ac4726af6e44c253dae532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:24:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43166
I20260620 23:24:34.476226 13677 tablet_service.cc:1514] Processing DeleteTablet for tablet c8efed6465ac4726af6e44c253dae532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:24:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50506
I20260620 23:24:34.476418 13790 tablet_replica.cc:330] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0: stopping tablet replica
I20260620 23:24:34.476774 13790 raft_consensus.cc:2229] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:24:34.477226 13790 raft_consensus.cc:2258] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:24:34.477893 13791 tablet_replica.cc:330] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c: stopping tablet replica
I20260620 23:24:34.478157 13792 tablet_replica.cc:330] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419: stopping tablet replica
I20260620 23:24:34.478327 13791 raft_consensus.cc:2229] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:24:34.478579 13792 raft_consensus.cc:2229] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:24:34.478806 13791 raft_consensus.cc:2258] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:24:34.479269 13792 raft_consensus.cc:2258] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:24:34.481259 13792 ts_tablet_manager.cc:1905] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:24:34.482213 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 13301
I20260620 23:24:34.482785 13791 ts_tablet_manager.cc:1905] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:24:34.484220 13790 ts_tablet_manager.cc:1905] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:24:34.488086 13792 ts_tablet_manager.cc:1918] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:24:34.488288 13792 log.cc:1196] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/c8efed6465ac4726af6e44c253dae532
I20260620 23:24:34.488831 13792 ts_tablet_manager.cc:1939] T c8efed6465ac4726af6e44c253dae532 P ad805516cf9640f6aa677624d8d01419: Deleting consensus metadata
I20260620 23:24:34.490437 13249 catalog_manager.cc:4841] TS ad805516cf9640f6aa677624d8d01419 (127.0.97.3:34935): tablet c8efed6465ac4726af6e44c253dae532 (table test-table [id=a745bf985d624fe0b3d28cf4e99012ae]) successfully deleted
I20260620 23:24:34.492698 13790 ts_tablet_manager.cc:1918] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:24:34.492873 13790 log.cc:1196] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/c8efed6465ac4726af6e44c253dae532
I20260620 23:24:34.493283 13790 ts_tablet_manager.cc:1939] T c8efed6465ac4726af6e44c253dae532 P df66dec349474b6ca25d285f36b955c0: Deleting consensus metadata
I20260620 23:24:34.493367 13791 ts_tablet_manager.cc:1918] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:24:34.494413 13791 log.cc:1196] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/c8efed6465ac4726af6e44c253dae532
I20260620 23:24:34.494850 13250 catalog_manager.cc:4841] TS df66dec349474b6ca25d285f36b955c0 (127.0.97.2:34453): tablet c8efed6465ac4726af6e44c253dae532 (table test-table [id=a745bf985d624fe0b3d28cf4e99012ae]) successfully deleted
I20260620 23:24:34.495306 13791 ts_tablet_manager.cc:1939] T c8efed6465ac4726af6e44c253dae532 P e50c1a955762468a85922df1c0ece60c: Deleting consensus metadata
I20260620 23:24:34.496963 13249 catalog_manager.cc:4841] TS e50c1a955762468a85922df1c0ece60c (127.0.97.1:45293): tablet c8efed6465ac4726af6e44c253dae532 (table test-table [id=a745bf985d624fe0b3d28cf4e99012ae]) successfully deleted
I20260620 23:24:34.499013 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.484425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.0.97.1:45293)
0620 23:24:34.484579 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:34.484584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:34.484601 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:24:34.484655 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:34.484659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:34.484731 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:34.484832 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:34.486428 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.487511 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.489322 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.489949 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.490077 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.492592 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:34.492605 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:34.492623 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:34.492658 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:34.496213 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:34.496969 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:34.496979 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:34.496986 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:34.497137 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:34.497544 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:34.497550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:34.497553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:34.497875 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:34.498162 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:34.498445 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 23:24:34.498639 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:24:34.499603 13455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:34.590018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 13301
I20260620 23:24:34.607044 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 13463
I20260620 23:24:34.622628 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.607913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.0.97.2:34453)
0620 23:24:34.608122 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:34.608128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:34.608183 (+ 55us) server_negotiation.cc:407] Connection header received
0620 23:24:34.608252 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:34.608261 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:34.608346 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:34.608461 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:34.610120 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.611377 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.612602 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.613221 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.613358 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.616101 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:34.616121 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:34.616147 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:34.616218 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:34.619898 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:34.620582 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:34.620592 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:34.620596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:34.620762 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:34.621082 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:34.621087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:34.621090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:34.621517 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:34.621818 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:34.622017 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 23:24:34.622231 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:24:34.623296 13600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:34.711799 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 13463
I20260620 23:24:34.734583 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 13609
I20260620 23:24:34.751134 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:34.735487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.0.97.3:34935)
0620 23:24:34.735682 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:34.735686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:34.735763 (+ 77us) server_negotiation.cc:407] Connection header received
0620 23:24:34.735956 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:34.735961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:34.736023 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:34.736117 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:34.738202 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.739125 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.741027 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:34.741524 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:34.741644 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:34.745071 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:34.745082 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:34.745096 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:34.745130 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:34.748166 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:34.749010 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:34.749016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:34.749018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:34.749152 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:34.749540 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:34.749543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:34.749544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:34.749842 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:34.750074 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:34.750394 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 23:24:34.750544 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:24:34.751859 13742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:34.869601 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 13609
I20260620 23:24:34.888633 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 13231
I20260620 23:24:34.889854 13294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:34.999143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 13231
I20260620 23:24:35.022845 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:24:35.358729 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:24:35.358955 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:24:35.704690 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:24:35.704 UTC [12284] LOG: received smart shutdown request
2026-06-20 23:24:35.707 UTC [12284] LOG: background worker "logical replication launcher" (PID 12293) exited with exit code 1
2026-06-20 23:24:35.708 UTC [12288] LOG: shutting down
2026-06-20 23:24:35.844 UTC [12284] LOG: database system is shut down
2026-06-20T23:24:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51064 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:24:36 dist-test-slave-5kzz krb5kdc[13805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:24:36 dist-test-slave-5kzz krb5kdc[13805](info): set up 2 sockets
Jun 20 23:24:36 dist-test-slave-5kzz krb5kdc[13805](info): commencing operation
krb5kdc: starting...
W20260620 23:24:37.026851 388 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:24:37 dist-test-slave-5kzz krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:24:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:24:37.233913 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38115
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38115,127.0.97.61:35071,127.0.97.60:44037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:24:37.554343 13818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:37.554617 13818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:37.554728 13818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:37.561771 13818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:24:37.561872 13818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:37.561918 13818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:37.561977 13818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:24:37.562021 13818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:24:37.571870 13818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38115,127.0.97.61:35071,127.0.97.60:44037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:37.573637 13818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:37.575511 13818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:37.588030 13823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:37.588140 13824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:37.588160 13826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:37.588856 13818 server_base.cc:1034] running on GCE node
I20260620 23:24:37.589547 13818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:37.590852 13818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:37.592067 13818 hybrid_clock.cc:648] HybridClock initialized: now 1781997877592034 us; error 43 us; skew 500 ppm
I20260620 23:24:37.592406 13818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:37 dist-test-slave-5kzz krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:37.595454 13818 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:24:37.596825 13818 webserver.cc:460] Webserver started at http://127.0.97.62:44161/ using document root <none> and password file <none>
I20260620 23:24:37.597468 13818 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:37.597622 13818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:37.597918 13818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:24:37.600790 13818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "5dce5b0a0d9f4eaca26108d3a723bc52"
format_stamp: "Formatted at 2026-06-20 23:24:37 on dist-test-slave-5kzz"
server_key: "743eebddf41424241f8114a560ee45b8"
server_key_iv: "de5d9acfbcf622507da3dff6d8a2b170"
server_key_version: "encryptionkey@0"
I20260620 23:24:37.601574 13818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "5dce5b0a0d9f4eaca26108d3a723bc52"
format_stamp: "Formatted at 2026-06-20 23:24:37 on dist-test-slave-5kzz"
server_key: "743eebddf41424241f8114a560ee45b8"
server_key_iv: "de5d9acfbcf622507da3dff6d8a2b170"
server_key_version: "encryptionkey@0"
I20260620 23:24:37.606472 13818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:24:37.609627 13833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:37.610913 13818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:24:37.611074 13818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "5dce5b0a0d9f4eaca26108d3a723bc52"
format_stamp: "Formatted at 2026-06-20 23:24:37 on dist-test-slave-5kzz"
server_key: "743eebddf41424241f8114a560ee45b8"
server_key_iv: "de5d9acfbcf622507da3dff6d8a2b170"
server_key_version: "encryptionkey@0"
I20260620 23:24:37.611279 13818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:37.622103 13818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:37.625762 13818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:37.626027 13818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:37.643873 13818 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38115
I20260620 23:24:37.643918 13884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38115 every 8 connection(s)
I20260620 23:24:37.645157 13818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:24:37.648653 13885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:37.655166 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 13818
I20260620 23:24:37.655377 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:24:37.654829 13885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:24:37.656059 388 external_mini_cluster.cc:1428] Setting key 5e14c1f7de3e0e0e35ab3e8f4ac46f92
Jun 20 23:24:37 dist-test-slave-5kzz krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260620 23:24:37.677915 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:37.657897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38115 (local address 127.0.0.1:59458)
0620 23:24:37.658762 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:37.658813 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:37.660729 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:37.661508 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:37.661513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:37.662192 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:37.662593 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:37.662619 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:37.664527 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:37.664531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:37.665546 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:37.665557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:37.666893 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:37.666898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:37.667030 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:37.667658 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:37.667678 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:37.669869 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:37.674130 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:37.674138 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:37.674147 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:37.674510 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:37.675004 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:37.675008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:37.675010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:37.675149 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:37.676090 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:37.676095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:37.676292 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 23:24:37.676481 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 23:24:37.676833 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":2,"thread_start_us":387,"threads_started":1}
I20260620 23:24:37.677945 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:37.657904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.0.97.62:38115)
0620 23:24:37.659579 (+ 1675us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:37.659586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:37.659637 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:24:37.661068 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:37.661073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:37.661302 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:37.661497 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:37.662813 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:37.664357 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:37.665686 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:37.666640 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:37.666866 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:37.670094 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:37.670106 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:37.670122 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:37.670161 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:37.673997 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:37.674635 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:37.674644 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:37.674648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:37.674858 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:37.675259 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:37.675266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:37.675272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:37.675960 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:37.676158 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:37.676672 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 23:24:37.677001 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":1457,"thread_start_us":1245,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 23:24:37.680866 13885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:24:37.681828 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:37.681464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54538)
0620 23:24:37.681625 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:37.681728 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:24:37.682031 13835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
I20260620 23:24:37.683516 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:37.683185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54554)
0620 23:24:37.683324 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:37.683408 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:24:37.683677 13885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35071: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 23:24:37.704085 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35071
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38115,127.0.97.61:35071,127.0.97.60:44037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:24:37.732993 13885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } attempt: 1
I20260620 23:24:37.734151 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:37.733699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54558)
0620 23:24:37.733913 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:37.734003 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 23:24:37.735270 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:37.734920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54566)
0620 23:24:37.735048 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:37.735131 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:24:37.735560 13885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35071: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
I20260620 23:24:37.803802 13885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } attempt: 2
I20260620 23:24:37.804971 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:37.804534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54582)
0620 23:24:37.804722 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:37.804814 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 23:24:37.806046 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:37.805715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54596)
0620 23:24:37.805838 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:37.805923 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:24:37.806277 13885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35071: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
I20260620 23:24:37.897650 13885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } attempt: 3
I20260620 23:24:37.898918 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:37.898433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54606)
0620 23:24:37.898637 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:37.898776 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:24:37.899943 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:37.899634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54618)
0620 23:24:37.899760 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:37.899822 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:24:37.900177 13885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35071: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
W20260620 23:24:38.023223 13892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:38.023499 13892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:38.023705 13892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:38.030650 13892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:24:38.030786 13892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:38.030884 13892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:38.030979 13892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:24:38.031059 13892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:24:38.040519 13892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38115,127.0.97.61:35071,127.0.97.60:44037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:38.041803 13892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:38.043179 13892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:24:38.043453 13885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } attempt: 4
I20260620 23:24:38.044674 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.044143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54622)
0620 23:24:38.044465 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.044549 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260620 23:24:38.045727 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.045360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54632)
0620 23:24:38.045510 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.045595 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 23:24:38.045917 13885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35071: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35071: connect: Connection refused (error 111)
W20260620 23:24:38.055514 13897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:38.055661 13898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:38.055536 13900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:38.060030 13892 server_base.cc:1034] running on GCE node
I20260620 23:24:38.060873 13892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:38.062208 13892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:38.063434 13892 hybrid_clock.cc:648] HybridClock initialized: now 1781997878063400 us; error 50 us; skew 500 ppm
I20260620 23:24:38.063779 13892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:38.067104 13892 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 23:24:38.068399 13892 webserver.cc:460] Webserver started at http://127.0.97.61:40957/ using document root <none> and password file <none>
I20260620 23:24:38.069017 13892 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:38.069108 13892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:38.069345 13892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:24:38.072158 13892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/data/instance:
uuid: "4be087ab7d2d4135aff3261d13953f7d"
format_stamp: "Formatted at 2026-06-20 23:24:38 on dist-test-slave-5kzz"
server_key: "b168a0d552ab3eb1d9f30fad418a8d85"
server_key_iv: "a0f366866904e37876e2f61443b4f080"
server_key_version: "encryptionkey@0"
I20260620 23:24:38.072862 13892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/wal/instance:
uuid: "4be087ab7d2d4135aff3261d13953f7d"
format_stamp: "Formatted at 2026-06-20 23:24:38 on dist-test-slave-5kzz"
server_key: "b168a0d552ab3eb1d9f30fad418a8d85"
server_key_iv: "a0f366866904e37876e2f61443b4f080"
server_key_version: "encryptionkey@0"
I20260620 23:24:38.077553 13892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 23:24:38.080637 13907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:38.081873 13892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:24:38.082055 13892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/wal
uuid: "4be087ab7d2d4135aff3261d13953f7d"
format_stamp: "Formatted at 2026-06-20 23:24:38 on dist-test-slave-5kzz"
server_key: "b168a0d552ab3eb1d9f30fad418a8d85"
server_key_iv: "a0f366866904e37876e2f61443b4f080"
server_key_version: "encryptionkey@0"
I20260620 23:24:38.082239 13892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:38.103880 13892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:38.107731 13892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:38.108011 13892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:38.125950 13892 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:35071
I20260620 23:24:38.125980 13958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:35071 every 8 connection(s)
I20260620 23:24:38.127321 13892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
I20260620 23:24:38.130836 13959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:38.136914 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 13892
I20260620 23:24:38.137143 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-1/wal/instance
I20260620 23:24:38.137550 388 external_mini_cluster.cc:1428] Setting key 9b428aff7881149bf3d925876ba0a7af
I20260620 23:24:38.136929 13959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 23:24:38.159811 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.139644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.0.97.62:38115)
0620 23:24:38.139854 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.139860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.141040 (+ 1180us) server_negotiation.cc:407] Connection header received
0620 23:24:38.143871 (+ 2831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.143875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.143953 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.144039 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.145694 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.146843 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.148358 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.149181 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.149323 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.152662 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.152671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.152679 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.152727 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.156678 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.157405 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.157412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.157417 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.157572 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.157969 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.157976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.157980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.158377 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.158543 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.159267 (+ 724us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.159524 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:24:38.160707 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.140041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38115 (local address 127.0.0.1:59472)
0620 23:24:38.140849 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.140911 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:38.143594 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:38.144337 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:38.144352 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:38.145000 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:38.145516 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.145531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.147019 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.147022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.148199 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.148210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.149380 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.149385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.149569 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.150255 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:38.150275 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:38.152431 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:38.156838 (+ 4407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.156844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.156857 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.157275 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.157689 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.157692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.157695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.157858 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.158492 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:38.158497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:38.158885 (+ 388us) client_negotiation.cc:769] Sending connection context
0620 23:24:38.159102 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 23:24:38.159548 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":320,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 23:24:38.162451 13959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:24:38.178383 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.163273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54644)
0620 23:24:38.163428 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.163442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:38.163520 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:38.164595 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:38.164602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:38.165115 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:38.165320 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.165334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.166607 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.166615 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.168165 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.168181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.169031 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.169043 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.169276 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.169928 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:38.169943 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:38.171805 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:38.176017 (+ 4212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.176021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.176024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.176342 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.176770 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.176774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.176776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.176932 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.177701 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:38.177704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:38.177821 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 23:24:38.177930 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:24:38.178116 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 23:24:38.178920 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.163563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.0.97.61:35071)
0620 23:24:38.164167 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.164173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.164196 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:24:38.164234 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.164239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.164436 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.164543 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.165482 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.166460 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.168370 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.168871 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.168988 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.172032 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.172047 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.172067 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.172111 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.175877 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.176470 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.176481 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.176487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.176627 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.177074 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.177082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.177087 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.177581 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.177795 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.178116 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.178403 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":352,"threads_started":1}
I20260620 23:24:38.181445 13959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:24:38.182492 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.182098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53788)
0620 23:24:38.182244 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.182362 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 23:24:38.182859 13910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
I20260620 23:24:38.184443 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.184068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53802)
0620 23:24:38.184267 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.184340 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:24:38.184665 13959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44037: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 23:24:38.190680 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44037
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38115,127.0.97.61:35071,127.0.97.60:44037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:24:38.233976 13959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } attempt: 1
I20260620 23:24:38.235288 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.234831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53808)
0620 23:24:38.235031 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.235140 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 23:24:38.236465 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.235931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53812)
0620 23:24:38.236161 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.236277 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 23:24:38.236697 13959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44037: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
I20260620 23:24:38.304979 13959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } attempt: 2
I20260620 23:24:38.306233 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.305717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53826)
0620 23:24:38.305941 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.306057 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:24:38.307238 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.306886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53828)
0620 23:24:38.307028 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.307107 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:24:38.307464 13959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44037: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
I20260620 23:24:38.345176 13885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } attempt: 5
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:24:38.359406 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.346185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.0.97.61:35071)
0620 23:24:38.346398 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.346406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.346429 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:24:38.346492 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.346504 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.346576 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.346704 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.347650 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.348287 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.349914 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.350353 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.350444 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.354053 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.354057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.354060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.354091 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.356702 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.357615 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.357620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.357622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.357738 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.358192 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.358195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.358197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.358447 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.358551 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.359025 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.359184 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:24:38.359601 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.345990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54652)
0620 23:24:38.346197 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.346220 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:38.346355 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:38.346770 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:38.346776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:38.347258 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:38.347497 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.347509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.348438 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.348443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.349775 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.349789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.350514 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.350520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.350748 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.351368 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:38.351391 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:38.353858 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:38.356871 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.356879 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.356885 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.357495 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.357859 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.357866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.357871 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.358064 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.358558 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:38.358564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:38.358770 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 23:24:38.358921 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 23:24:38.359236 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:24:38.360754 13885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:24:38.361855 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.361421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53836)
0620 23:24:38.361603 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.361692 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 23:24:38.362843 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.362448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53850)
0620 23:24:38.362598 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.362728 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:24:38.363063 13885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44037: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
I20260620 23:24:38.398715 13959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } attempt: 3
I20260620 23:24:38.399893 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.399378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53852)
0620 23:24:38.399608 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.399712 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 23:24:38.401054 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.400530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53854)
0620 23:24:38.400779 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.400885 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260620 23:24:38.401269 13959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44037: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
I20260620 23:24:38.414294 13885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } attempt: 1
I20260620 23:24:38.415577 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.415032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53864)
0620 23:24:38.415256 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.415362 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 23:24:38.416625 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.416216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53876)
0620 23:24:38.416382 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.416480 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:24:38.416833 13885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44037: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
I20260620 23:24:38.485141 13885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } attempt: 2
I20260620 23:24:38.486380 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.485828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53884)
0620 23:24:38.486046 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.486189 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 23:24:38.487519 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.487101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53894)
0620 23:24:38.487271 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.487363 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 23:24:38.487723 13885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44037: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
W20260620 23:24:38.511649 13966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:38.511955 13966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:38.512128 13966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:38.519191 13966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:24:38.519320 13966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:38.519418 13966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:38.519510 13966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:24:38.519589 13966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:24:38.530439 13966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38115,127.0.97.61:35071,127.0.97.60:44037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:38.531764 13966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:38.533087 13966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:38.543985 13971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:38.544034 13972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:38.544059 13974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:38.544510 13959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } attempt: 4
I20260620 23:24:38.545198 13966 server_base.cc:1034] running on GCE node
I20260620 23:24:38.545557 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.545151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53898)
0620 23:24:38.545331 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.545421 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 23:24:38.546103 13966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:38.546461 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.546144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53908)
0620 23:24:38.546286 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.546361 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 23:24:38.546665 13959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44037: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
I20260620 23:24:38.547549 13966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:38.548774 13966 hybrid_clock.cc:648] HybridClock initialized: now 1781997878548728 us; error 54 us; skew 500 ppm
I20260620 23:24:38.549108 13966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:38.552661 13966 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 23:24:38.554638 13966 webserver.cc:460] Webserver started at http://127.0.97.60:45577/ using document root <none> and password file <none>
I20260620 23:24:38.555347 13966 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:38.555485 13966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:38.555797 13966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:24:38.558646 13966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/data/instance:
uuid: "ec2ee6b3fa644452ae5577b85c170c9a"
format_stamp: "Formatted at 2026-06-20 23:24:38 on dist-test-slave-5kzz"
server_key: "584f0403554e45761e5e5fa1af7ff47b"
server_key_iv: "e8491f1fb545637263261703a3bef873"
server_key_version: "encryptionkey@0"
I20260620 23:24:38.559443 13966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/wal/instance:
uuid: "ec2ee6b3fa644452ae5577b85c170c9a"
format_stamp: "Formatted at 2026-06-20 23:24:38 on dist-test-slave-5kzz"
server_key: "584f0403554e45761e5e5fa1af7ff47b"
server_key_iv: "e8491f1fb545637263261703a3bef873"
server_key_version: "encryptionkey@0"
I20260620 23:24:38.564244 13966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:24:38.567483 13981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:38.568703 13966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:24:38.568878 13966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/wal
uuid: "ec2ee6b3fa644452ae5577b85c170c9a"
format_stamp: "Formatted at 2026-06-20 23:24:38 on dist-test-slave-5kzz"
server_key: "584f0403554e45761e5e5fa1af7ff47b"
server_key_iv: "e8491f1fb545637263261703a3bef873"
server_key_version: "encryptionkey@0"
I20260620 23:24:38.569080 13966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:38.586081 13966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:38.590343 13966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:38.590622 13966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:38.594002 13885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } attempt: 3
I20260620 23:24:38.595191 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.594590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53922)
0620 23:24:38.594872 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.594980 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 23:24:38.596395 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.595835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53926)
0620 23:24:38.596145 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.596262 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260620 23:24:38.596657 13885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44037: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44037: connect: Connection refused (error 111)
I20260620 23:24:38.608548 13966 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44037
I20260620 23:24:38.608577 14032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44037 every 8 connection(s)
I20260620 23:24:38.609861 13966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
I20260620 23:24:38.611213 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 13966
I20260620 23:24:38.611414 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997680534866-388-0/minicluster-data/master-2/wal/instance
I20260620 23:24:38.611843 388 external_mini_cluster.cc:1428] Setting key 72652e297f646f5c3474758b8555de51
I20260620 23:24:38.613826 14033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:38.621577 14033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:38.635308 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.613606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.0.97.60:44037)
0620 23:24:38.614327 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.614341 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.614386 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:24:38.616614 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.616639 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.617124 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.617526 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.618847 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.620135 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.621423 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.622115 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.622313 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.626784 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.626795 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.626808 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.626860 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.630358 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.631109 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.631114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.631121 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.631278 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.632237 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.632244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.632247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.632791 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.633085 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.633443 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.633857 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":371,"threads_started":1}
I20260620 23:24:38.638926 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.624102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38115 (local address 127.0.0.1:59476)
0620 23:24:38.624776 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.624814 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:38.624991 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:38.625330 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:38.625336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:38.625896 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:38.626165 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.626172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.627453 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.627459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.628664 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.628675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.629639 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.629643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.629757 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.630392 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:38.630432 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:38.632701 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:38.636400 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.636409 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.636414 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.636810 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.637186 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.637189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.637191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.637319 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.637843 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:38.637850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:38.638024 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 23:24:38.638194 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 23:24:38.638492 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"spinlock_wait_cycles":14720,"thread_start_us":374,"threads_started":1}
I20260620 23:24:38.638924 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.624080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.0.97.62:38115)
0620 23:24:38.624244 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.624248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.624909 (+ 661us) server_negotiation.cc:407] Connection header received
0620 23:24:38.625111 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.625117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.625183 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.625280 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.626305 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.627282 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.628805 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.629470 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.629586 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.632888 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.632894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.632898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.632936 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.636262 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.636928 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.636933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.636938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.637077 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.637422 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.637425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.637428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.637722 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.637880 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.638297 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.638499 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:24:38.640638 14033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:24:38.655035 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.641255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35071 (local address 127.0.0.1:54664)
0620 23:24:38.641452 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.641469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:38.641558 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:38.641992 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:38.641995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:38.642363 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:38.642540 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.642552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.643753 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.643758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.645083 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.645097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.646119 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.646125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.646337 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.646925 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:38.646944 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:38.649145 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:38.652406 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.652411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.652417 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.652769 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.653194 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.653201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.653206 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.653359 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.654013 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:38.654021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:38.654169 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 23:24:38.654352 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 23:24:38.654628 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 23:24:38.655066 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.641505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.0.97.61:35071)
0620 23:24:38.641716 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.641723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.641742 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:24:38.641796 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.641801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.641873 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.641976 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.642742 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.643584 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.645252 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.645961 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.646100 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.649323 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.649328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.649331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.649360 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.652275 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.652894 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.652901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.652904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.653045 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.653531 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.653536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.653540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.653867 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.654020 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.654457 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.654666 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:24:38.655599 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.638447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.0.97.62:38115)
0620 23:24:38.639054 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.639060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.639126 (+ 66us) server_negotiation.cc:407] Connection header received
0620 23:24:38.639308 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.639312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.639374 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.639484 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.640900 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.642137 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.643659 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.644099 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.644242 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.648769 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.648775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.648780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.648814 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.652354 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.653150 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.653158 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.653162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.653322 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.653735 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.653740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.653742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.654098 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.654380 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.654706 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.654920 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":293,"threads_started":1}
I20260620 23:24:38.656599 14033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:24:38.670223 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.657192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53938)
0620 23:24:38.657349 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.657364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:38.657449 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:38.657985 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:38.657992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:38.658448 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:38.658652 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.658667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.659843 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.659847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.661066 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.661080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.661848 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.661852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.661990 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.662552 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:38.662588 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:38.664855 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:38.667800 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.667805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.667809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.668311 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.668684 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.668689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.668694 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.668855 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.669396 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:38.669399 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:38.669506 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 23:24:38.669618 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 23:24:38.669792 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 23:24:38.670296 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.657586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.0.97.60:44037)
0620 23:24:38.657733 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.657738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.657753 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:24:38.657793 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.657798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.657856 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.657928 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.658803 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.659724 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.661196 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.661702 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.661816 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.665045 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.665052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.665059 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.665098 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.667660 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.668437 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.668440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.668442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.668559 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.668971 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.668974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.668975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.669256 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.669416 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.669739 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.669969 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:24:38.671978 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.657110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.0.97.61:35071)
0620 23:24:38.657271 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.657276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.657294 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:24:38.657493 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.657500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.657570 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.657673 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.659109 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.660024 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.661550 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.662099 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.662238 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.666421 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.666425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.666428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.666454 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.669224 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.669885 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.669890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.669894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.670032 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.670482 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.670490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.670495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.670872 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.671098 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.671437 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.671646 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:24:38.675854 14033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a: Bootstrap starting.
I20260620 23:24:38.676688 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:24:38.679041 14033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:38.680183 14033 log.cc:824] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:38.683458 14033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a: No bootstrap required, opened a new log
I20260620 23:24:38.695742 14033 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } }
I20260620 23:24:38.696353 14033 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:38.696521 14033 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2ee6b3fa644452ae5577b85c170c9a, State: Initialized, Role: FOLLOWER
I20260620 23:24:38.697412 14033 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } }
I20260620 23:24:38.700701 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.681341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.0.97.61:35071)
0620 23:24:38.681507 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.681513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.682416 (+ 903us) server_negotiation.cc:407] Connection header received
0620 23:24:38.682723 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.682728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.682798 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.682867 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.685316 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.686425 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.688601 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.689192 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.689284 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.693221 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.693227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.693232 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.693290 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.696773 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.697582 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.697590 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.697594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.697742 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.698268 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.698276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.698282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.698821 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.699101 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.699885 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.700186 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:24:38.701797 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.681651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.0.97.62:38115)
0620 23:24:38.681852 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.681857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.682156 (+ 299us) server_negotiation.cc:407] Connection header received
0620 23:24:38.682435 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.682441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.682537 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.682632 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.685581 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.686975 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.689755 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.690600 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.690742 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.694653 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.694659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.694664 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.694729 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.698426 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.699455 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.699462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.699466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.699612 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.700108 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.700114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.700119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.700480 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.700695 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.701205 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.701447 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:24:38.705247 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } } }
I20260620 23:24:38.705600 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:24:38.707134 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.682818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.0.97.60:44037)
0620 23:24:38.683112 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.683118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.691207 (+ 8089us) server_negotiation.cc:407] Connection header received
0620 23:24:38.691424 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.691429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.691511 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.691615 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.693753 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.694892 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.696929 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.697659 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.697755 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.701150 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.701156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.701160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.701193 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.704515 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.705161 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.705166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.705169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.705315 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.705734 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.705741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.705744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.706057 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.706277 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.706543 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.706788 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260620 23:24:38.707501 14033 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:24:38.717847 14058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:24:38.717960 14058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:24:38.714792 13966 master_runner.cc:186] Error getting master registration for 127.0.97.62:38115: OK, instance_id { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" instance_seqno: 1781997877638713 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:24:38.723287 13966 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:24:38.764602 14047 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:24:38.764894 14047 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } }
I20260620 23:24:38.766271 14047 leader_election.cc:288] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dce5b0a0d9f4eaca26108d3a723bc52 (127.0.97.62:38115), 4be087ab7d2d4135aff3261d13953f7d (127.0.97.61:35071)
I20260620 23:24:38.773862 13885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } attempt: 4
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:24:38.790799 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.774407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53962)
0620 23:24:38.774603 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.774624 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:38.774751 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:38.775089 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:38.775094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:38.775485 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:38.775632 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.775640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.777097 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.777103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.778551 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.778565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.780412 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.780418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.780607 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.781270 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:38.781293 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:38.784141 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:38.788044 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.788049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.788053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.788557 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.788956 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.788961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.788965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.789118 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.789715 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:38.789719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:38.789876 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 23:24:38.790029 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 23:24:38.790359 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 23:24:38.791952 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.774559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.0.97.60:44037)
0620 23:24:38.774731 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.774736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.774754 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:24:38.774889 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.774894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.774971 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.775058 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.775768 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.776945 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.778723 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.779416 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.779558 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.784305 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.784312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.784317 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.784353 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.787908 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.788673 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.788680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.788684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.788843 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.789246 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.789251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.789254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.789603 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.789735 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.791178 (+ 1443us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.791597 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 23:24:38.797749 13885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52: Bootstrap starting.
I20260620 23:24:38.801761 13885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:38.803120 13885 log.cc:824] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:38.806269 13885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52: No bootstrap required, opened a new log
I20260620 23:24:38.816562 13885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } }
I20260620 23:24:38.817001 13885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:38.817114 13885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dce5b0a0d9f4eaca26108d3a723bc52, State: Initialized, Role: FOLLOWER
I20260620 23:24:38.817739 13885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } }
I20260620 23:24:38.818650 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } } }
I20260620 23:24:38.819051 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:24:38.820034 13885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:24:38.820605 13860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" is_pre_election: true
I20260620 23:24:38.821319 13860 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec2ee6b3fa644452ae5577b85c170c9a in term 0.
I20260620 23:24:38.825018 13984 leader_election.cc:302] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dce5b0a0d9f4eaca26108d3a723bc52, ec2ee6b3fa644452ae5577b85c170c9a; no voters:
I20260620 23:24:38.825495 14047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:24:38.825641 14047 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:24:38.825740 14047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:38.829090 14047 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } }
I20260620 23:24:38.829869 14047 leader_election.cc:288] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [CANDIDATE]: Term 1 election: Requested vote from peers 5dce5b0a0d9f4eaca26108d3a723bc52 (127.0.97.62:38115), 4be087ab7d2d4135aff3261d13953f7d (127.0.97.61:35071)
I20260620 23:24:38.830180 13860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52"
I20260620 23:24:38.830469 13860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:38.834705 13860 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec2ee6b3fa644452ae5577b85c170c9a in term 1.
I20260620 23:24:38.835273 13984 leader_election.cc:302] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dce5b0a0d9f4eaca26108d3a723bc52, ec2ee6b3fa644452ae5577b85c170c9a; no voters:
I20260620 23:24:38.835638 14047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:24:38.836236 14047 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [term 1 LEADER]: Becoming Leader. State: Replica: ec2ee6b3fa644452ae5577b85c170c9a, State: Running, Role: LEADER
I20260620 23:24:38.836793 14047 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } }
W20260620 23:24:38.842572 13987 tablet.cc:2369] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:24:38.845901 13959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } attempt: 5
Jun 20 23:24:38 dist-test-slave-5kzz krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:24:38.856772 14067 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec2ee6b3fa644452ae5577b85c170c9a. Latest consensus state: current_term: 1 leader_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } } }
I20260620 23:24:38.857167 14067 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:38.860127 14075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:24:38.849975 13818 master_runner.cc:186] Error getting master registration for 127.0.97.61:35071: OK, instance_id { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" instance_seqno: 1781997878120973 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:24:38.861994 13818 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:24:38.864420 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.846419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44037 (local address 127.0.0.1:53978)
0620 23:24:38.846769 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:38.846789 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:38.846890 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:38.847182 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:38.847199 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:38.847713 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:38.847914 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.847926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.849652 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.849659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.851126 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:38.851139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.852647 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.852653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:38.852844 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.853545 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:38.853569 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:38.856742 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:38.861571 (+ 4829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.861577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.861582 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.862105 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.862575 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:38.862579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:38.862583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:38.862777 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:38.863426 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:38.863431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:38.863574 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 23:24:38.863704 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 23:24:38.864021 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260620 23:24:38.864487 14075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:24:38.865566 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:38.846761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53978 (local address 127.0.97.60:44037)
0620 23:24:38.846918 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:38.846925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:38.846944 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:24:38.846999 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:38.847005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:38.847075 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:38.847159 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:38.848346 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.849498 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.851605 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:38.852314 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:38.852445 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:38.857033 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:38.857040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:38.857046 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:38.857088 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:38.861396 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.862275 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.862280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.862283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.862434 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:38.862917 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:38.862922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:38.862928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:38.863289 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:38.864796 (+ 1507us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:38.864927 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 23:24:38.865185 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:24:38.871125 13959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d: Bootstrap starting.
W20260620 23:24:38.874389 14074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:24:38.874511 14074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:24:38.875139 13959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:38.876434 13959 log.cc:824] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d: Log is configured to *not* fsync() on all Append() calls
W20260620 23:24:38.878718 13839 tablet.cc:2369] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:24:38.889917 13959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d: No bootstrap required, opened a new log
I20260620 23:24:38.891919 13860 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [term 1 FOLLOWER]: Refusing update from remote peer ec2ee6b3fa644452ae5577b85c170c9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:24:38.896119 14047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:24:38.907501 13959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } }
I20260620 23:24:38.908152 13959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:38.908331 13959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be087ab7d2d4135aff3261d13953f7d, State: Initialized, Role: FOLLOWER
I20260620 23:24:38.909430 13959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } }
I20260620 23:24:38.911000 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } } }
I20260620 23:24:38.911962 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:24:38.912909 13933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4be087ab7d2d4135aff3261d13953f7d"
I20260620 23:24:38.913169 13932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:38.914294 13959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:24:38.917538 13932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [term 1 FOLLOWER]: Refusing update from remote peer ec2ee6b3fa644452ae5577b85c170c9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:24:38.919679 14047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:24:38.919996 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec2ee6b3fa644452ae5577b85c170c9a. Latest consensus state: current_term: 1 leader_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } } }
I20260620 23:24:38.920258 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:24:38.921921 13934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4be087ab7d2d4135aff3261d13953f7d" is_pre_election: true
I20260620 23:24:38.922314 13934 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec2ee6b3fa644452ae5577b85c170c9a in term 1.
I20260620 23:24:38.925588 14077 mvcc.cc:205] Tried to move back new op lower bound from 7299063311928463360 to 7299063311720476672. Current Snapshot: MvccSnapshot[applied={T|T < 7299063311928463360}]
I20260620 23:24:38.931583 14079 mvcc.cc:205] Tried to move back new op lower bound from 7299063311928463360 to 7299063311720476672. Current Snapshot: MvccSnapshot[applied={T|T < 7299063311928463360}]
I20260620 23:24:38.932025 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } } }
I20260620 23:24:38.932266 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:24:38.940128 14067 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } } }
I20260620 23:24:38.940505 14067 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:38.941603 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec2ee6b3fa644452ae5577b85c170c9a. Latest consensus state: current_term: 1 leader_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } } }
I20260620 23:24:38.941880 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:24:38.947185 14067 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } } }
I20260620 23:24:38.947443 14067 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:38.949023 14075 catalog_manager.cc:1347] Generated new cluster ID: e6bcb81fdcaf4af3b6d93da2d2c9ab4a
I20260620 23:24:38.949106 14075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:24:38.953704 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce5b0a0d9f4eaca26108d3a723bc52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38115 } } peers { permanent_uuid: "4be087ab7d2d4135aff3261d13953f7d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35071 } } peers { permanent_uuid: "ec2ee6b3fa644452ae5577b85c170c9a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44037 } } }
I20260620 23:24:38.954015 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:24:38.971496 14097 catalog_manager.cc:1259] Loaded cluster ID: e6bcb81fdcaf4af3b6d93da2d2c9ab4a
I20260620 23:24:38.971598 14097 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d: loading cluster ID for follower catalog manager: success
W20260620 23:24:38.972812 14097 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 23:24:38.972939 14097 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:24:38.990056 14075 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:24:38.992007 14075 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:24:39.013849 14075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec2ee6b3fa644452ae5577b85c170c9a: Generated new TSK 0
I20260620 23:24:39.015117 14075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:24:39.876616 14074 catalog_manager.cc:1259] Loaded cluster ID: e6bcb81fdcaf4af3b6d93da2d2c9ab4a
I20260620 23:24:39.876760 14074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52: loading cluster ID for follower catalog manager: success
I20260620 23:24:39.879441 14074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52: acquiring CA information for follower catalog manager: success
I20260620 23:24:39.881138 14074 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5dce5b0a0d9f4eaca26108d3a723bc52: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:24:39.976310 14097 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d: acquiring CA information for follower catalog manager: success
I20260620 23:24:39.978161 14097 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4be087ab7d2d4135aff3261d13953f7d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:24:40.150604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 13818
I20260620 23:24:40.151857 13880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:40.259131 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 13818
I20260620 23:24:40.279012 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 13892
I20260620 23:24:40.280027 13954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:40.381948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 13892
I20260620 23:24:40.401250 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 13966
I20260620 23:24:40.402307 14028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:40.506294 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 13966
2026-06-20T23:24:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4542 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:24:40 dist-test-slave-5kzz krb5kdc[14110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:24:40 dist-test-slave-5kzz krb5kdc[14110](info): set up 2 sockets
Jun 20 23:24:40 dist-test-slave-5kzz krb5kdc[14110](info): commencing operation
krb5kdc: starting...
W20260620 23:24:41.567415 388 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:24:41 dist-test-slave-5kzz krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:24:41.784127 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43093
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:24:42.102169 14123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:42.102464 14123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:42.102553 14123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:42.109246 14123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:24:42.109346 14123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:42.109390 14123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:42.109445 14123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:24:42.109484 14123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:24:42.118637 14123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:42.119901 14123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:42.121095 14123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:42.132151 14128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:42.132185 14131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:42.132236 14129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:42.133705 14123 server_base.cc:1034] running on GCE node
I20260620 23:24:42.134375 14123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:42.135739 14123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:42.136929 14123 hybrid_clock.cc:648] HybridClock initialized: now 1781997882136876 us; error 51 us; skew 500 ppm
I20260620 23:24:42.137216 14123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:42 dist-test-slave-5kzz krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:42.140321 14123 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:24:42.141713 14123 webserver.cc:460] Webserver started at http://127.0.97.62:39155/ using document root <none> and password file <none>
I20260620 23:24:42.142328 14123 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:42.142431 14123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:42.142670 14123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:24:42.145609 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "231de26e927a4b0d97ed84aec83cad31"
format_stamp: "Formatted at 2026-06-20 23:24:42 on dist-test-slave-5kzz"
server_key: "66a55a1e1e41aa38bf5656cb77ec7d17"
server_key_iv: "a92d64dbef7888314d16ddda9bd8a50e"
server_key_version: "encryptionkey@0"
I20260620 23:24:42.146299 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "231de26e927a4b0d97ed84aec83cad31"
format_stamp: "Formatted at 2026-06-20 23:24:42 on dist-test-slave-5kzz"
server_key: "66a55a1e1e41aa38bf5656cb77ec7d17"
server_key_iv: "a92d64dbef7888314d16ddda9bd8a50e"
server_key_version: "encryptionkey@0"
I20260620 23:24:42.151710 14123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 23:24:42.155031 14138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:42.156284 14123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:24:42.156466 14123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "231de26e927a4b0d97ed84aec83cad31"
format_stamp: "Formatted at 2026-06-20 23:24:42 on dist-test-slave-5kzz"
server_key: "66a55a1e1e41aa38bf5656cb77ec7d17"
server_key_iv: "a92d64dbef7888314d16ddda9bd8a50e"
server_key_version: "encryptionkey@0"
I20260620 23:24:42.156656 14123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:42.183434 14123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:42.187568 14123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:42.187849 14123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:42.205788 14123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43093
I20260620 23:24:42.205821 14189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43093 every 8 connection(s)
I20260620 23:24:42.207187 14123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:24:42.210772 14190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:42.215236 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 14123
I20260620 23:24:42.215453 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:24:42.216290 388 external_mini_cluster.cc:1428] Setting key 4c8f7034346b8012957c7ce15dc6573d
I20260620 23:24:42.223443 14190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31: Bootstrap starting.
I20260620 23:24:42.227137 14190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:42.228377 14190 log.cc:824] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:42.231318 14190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31: No bootstrap required, opened a new log
Jun 20 23:24:42 dist-test-slave-5kzz krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:42.241744 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:42.218092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.0.97.62:43093)
0620 23:24:42.218893 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:42.218910 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:42.218972 (+ 62us) server_negotiation.cc:407] Connection header received
0620 23:24:42.221725 (+ 2753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:42.221753 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:42.222221 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:42.222650 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:42.224048 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:42.226042 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:42.227521 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:42.228575 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:42.228747 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:42.232997 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:42.233008 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:42.233031 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:42.233079 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:42.237197 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:42.237910 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:42.237919 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:42.237927 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:42.238127 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:42.238500 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:42.238505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:42.238509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:42.239231 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:42.239547 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:42.239886 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 23:24:42.240349 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":320,"threads_started":1}
I20260620 23:24:42.242095 14190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231de26e927a4b0d97ed84aec83cad31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43093 } }
I20260620 23:24:42.242484 14190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:42.242604 14190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231de26e927a4b0d97ed84aec83cad31, State: Initialized, Role: FOLLOWER
I20260620 23:24:42.243302 14190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231de26e927a4b0d97ed84aec83cad31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43093 } }
I20260620 23:24:42.243541 14190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:24:42.243696 14190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:24:42.243887 14190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:42.245934 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:24:42.246579 14190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231de26e927a4b0d97ed84aec83cad31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43093 } }
I20260620 23:24:42.247152 14190 leader_election.cc:302] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 231de26e927a4b0d97ed84aec83cad31; no voters:
I20260620 23:24:42.247807 14190 leader_election.cc:288] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:24:42.247915 14195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:24:42.249401 14195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [term 1 LEADER]: Becoming Leader. State: Replica: 231de26e927a4b0d97ed84aec83cad31, State: Running, Role: LEADER
I20260620 23:24:42.249970 14195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231de26e927a4b0d97ed84aec83cad31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43093 } }
I20260620 23:24:42.250351 14190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:24:42.252795 14197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 231de26e927a4b0d97ed84aec83cad31. Latest consensus state: current_term: 1 leader_uuid: "231de26e927a4b0d97ed84aec83cad31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231de26e927a4b0d97ed84aec83cad31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43093 } } }
I20260620 23:24:42.253541 14197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:42.253249 14196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "231de26e927a4b0d97ed84aec83cad31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231de26e927a4b0d97ed84aec83cad31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43093 } } }
I20260620 23:24:42.254024 14196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31 [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:42.254459 14204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:24:42.259967 14204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:24:42.268393 14204 catalog_manager.cc:1347] Generated new cluster ID: 10e5c6b4c9964e188e573af644363abb
I20260620 23:24:42.268525 14204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:24:42.288087 14204 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:24:42.289354 14204 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:24:42.298802 14204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 231de26e927a4b0d97ed84aec83cad31: Generated new TSK 0
I20260620 23:24:42.299603 14204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:24:42.360217 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 14123
I20260620 23:24:42.361441 14185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:42.456367 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 14123
2026-06-20T23:24:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1946 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:24:42 dist-test-slave-5kzz krb5kdc[14218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:24:42 dist-test-slave-5kzz krb5kdc[14218](info): set up 2 sockets
Jun 20 23:24:42 dist-test-slave-5kzz krb5kdc[14218](info): commencing operation
krb5kdc: starting...
W20260620 23:24:43.523144 388 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:24:43 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:24:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:24:43.741783 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46597
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:24:44.058117 14231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:44.058382 14231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:44.058570 14231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:44.066078 14231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:24:44.066164 14231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:44.066283 14231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:44.066372 14231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:24:44.066435 14231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:24:44.075742 14231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:44.076948 14231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:44.078437 14231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:44.090346 14236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:44.090322 14237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:44.090395 14239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:44.091168 14231 server_base.cc:1034] running on GCE node
I20260620 23:24:44.091929 14231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:44.093256 14231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:44.094501 14231 hybrid_clock.cc:648] HybridClock initialized: now 1781997884094447 us; error 61 us; skew 500 ppm
I20260620 23:24:44.094971 14231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:44 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:44.097844 14231 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:24:44.099232 14231 webserver.cc:460] Webserver started at http://127.0.97.62:42747/ using document root <none> and password file <none>
I20260620 23:24:44.099871 14231 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:44.100018 14231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:44.100306 14231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:24:44.103207 14231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "2596abde319b46c7a2b06674a8c3f2c9"
format_stamp: "Formatted at 2026-06-20 23:24:44 on dist-test-slave-5kzz"
server_key: "368bb5da9634f96db0e56532ddbdbf65"
server_key_iv: "f65942ebe855af550b48199d526a23b4"
server_key_version: "encryptionkey@0"
I20260620 23:24:44.103941 14231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "2596abde319b46c7a2b06674a8c3f2c9"
format_stamp: "Formatted at 2026-06-20 23:24:44 on dist-test-slave-5kzz"
server_key: "368bb5da9634f96db0e56532ddbdbf65"
server_key_iv: "f65942ebe855af550b48199d526a23b4"
server_key_version: "encryptionkey@0"
I20260620 23:24:44.108650 14231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:24:44.111829 14246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:44.113075 14231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:24:44.113256 14231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "2596abde319b46c7a2b06674a8c3f2c9"
format_stamp: "Formatted at 2026-06-20 23:24:44 on dist-test-slave-5kzz"
server_key: "368bb5da9634f96db0e56532ddbdbf65"
server_key_iv: "f65942ebe855af550b48199d526a23b4"
server_key_version: "encryptionkey@0"
I20260620 23:24:44.113435 14231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:44.135819 14231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:44.139437 14231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:44.139703 14231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:44.156883 14231 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46597
I20260620 23:24:44.156924 14297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46597 every 8 connection(s)
I20260620 23:24:44.158334 14231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:24:44.161873 14298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:44.163022 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 14231
I20260620 23:24:44.163239 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:24:44.163838 388 external_mini_cluster.cc:1428] Setting key 1ca19ff0bc1ed3479acf4f18f797954f
I20260620 23:24:44.175355 14298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Bootstrap starting.
Jun 20 23:24:44 dist-test-slave-5kzz krb5kdc[14218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:44.179054 14298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:44.180132 14298 log.cc:824] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:44.183162 14298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: No bootstrap required, opened a new log
I20260620 23:24:44.186394 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:44.165700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.0.97.62:46597)
0620 23:24:44.166524 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:44.166535 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:44.166575 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:24:44.168098 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:44.168121 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:44.168499 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:44.168793 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:44.170544 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:44.171846 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:44.173933 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:44.174550 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:44.174734 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:44.179470 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:44.179477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:44.179493 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:44.179531 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:44.182441 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:44.183358 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:44.183364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:44.183370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:44.183544 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:44.183963 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:44.183966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:44.183969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:44.184413 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:44.184630 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:44.185068 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 23:24:44.185414 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":415,"threads_started":1}
I20260620 23:24:44.191157 14298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } }
I20260620 23:24:44.191485 14298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:44.191603 14298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2596abde319b46c7a2b06674a8c3f2c9, State: Initialized, Role: FOLLOWER
I20260620 23:24:44.192123 14298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } }
I20260620 23:24:44.192345 14298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:24:44.192487 14298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:24:44.192655 14298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:44.194634 14298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } }
I20260620 23:24:44.195279 14298 leader_election.cc:302] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2596abde319b46c7a2b06674a8c3f2c9; no voters:
I20260620 23:24:44.195837 14298 leader_election.cc:288] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:24:44.195964 14303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:24:44.196612 14303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 1 LEADER]: Becoming Leader. State: Replica: 2596abde319b46c7a2b06674a8c3f2c9, State: Running, Role: LEADER
I20260620 23:24:44.197134 14303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } }
I20260620 23:24:44.197753 14298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:24:44.199514 14305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2596abde319b46c7a2b06674a8c3f2c9. Latest consensus state: current_term: 1 leader_uuid: "2596abde319b46c7a2b06674a8c3f2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } } }
I20260620 23:24:44.199837 14305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:44.201252 14304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2596abde319b46c7a2b06674a8c3f2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } } }
I20260620 23:24:44.201545 14304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:44.201555 14308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:24:44.207414 14308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:24:44.215790 14308 catalog_manager.cc:1347] Generated new cluster ID: 3401ee2860bd4a168eeec4ab5b478eff
I20260620 23:24:44.215898 14308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:24:44.230226 14308 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:24:44.231529 14308 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:24:44.239214 14308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Generated new TSK 0
I20260620 23:24:44.240170 14308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:24:44.302757 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46597
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:24:44.620278 14324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:44.620573 14324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:44.620735 14324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:44.627723 14324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:44.627861 14324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:44.628038 14324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:24:44.637208 14324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:44.638361 14324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:44.639626 14324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:44.652328 14332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:44.652362 14329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:44.652418 14330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:44.653414 14324 server_base.cc:1034] running on GCE node
I20260620 23:24:44.654021 14324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:44.654942 14324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:44.656172 14324 hybrid_clock.cc:648] HybridClock initialized: now 1781997884656119 us; error 46 us; skew 500 ppm
I20260620 23:24:44.656478 14324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:44 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:44.659554 14324 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:24:44.660930 14324 webserver.cc:460] Webserver started at http://127.0.97.1:36769/ using document root <none> and password file <none>
I20260620 23:24:44.661580 14324 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:44.661721 14324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:44.662045 14324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:24:44.665040 14324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "a3b925c788004b839a529b99cdca8a3b"
format_stamp: "Formatted at 2026-06-20 23:24:44 on dist-test-slave-5kzz"
server_key: "ae8c8aee91e618a594e0cd31b5f9be94"
server_key_iv: "231f70efbe16e01bc54f5c30f201f54d"
server_key_version: "encryptionkey@0"
I20260620 23:24:44.665799 14324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a3b925c788004b839a529b99cdca8a3b"
format_stamp: "Formatted at 2026-06-20 23:24:44 on dist-test-slave-5kzz"
server_key: "ae8c8aee91e618a594e0cd31b5f9be94"
server_key_iv: "231f70efbe16e01bc54f5c30f201f54d"
server_key_version: "encryptionkey@0"
I20260620 23:24:44.670547 14324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 23:24:44.673842 14339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:44.675112 14324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:24:44.675284 14324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "a3b925c788004b839a529b99cdca8a3b"
format_stamp: "Formatted at 2026-06-20 23:24:44 on dist-test-slave-5kzz"
server_key: "ae8c8aee91e618a594e0cd31b5f9be94"
server_key_iv: "231f70efbe16e01bc54f5c30f201f54d"
server_key_version: "encryptionkey@0"
I20260620 23:24:44.675478 14324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:44.701944 14324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:44.705940 14324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:44.706214 14324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:44.707124 14324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:44.708420 14324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:44.708513 14324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:44.708649 14324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:44.708715 14324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:44.741873 14324 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40321
I20260620 23:24:44.741981 14451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40321 every 8 connection(s)
I20260620 23:24:44.743187 14324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:24:44.745625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 14324
I20260620 23:24:44.745848 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:24:44.746496 388 external_mini_cluster.cc:1428] Setting key 84a6a0c4bbcc328fbecae71b9fd394be
Jun 20 23:24:44 dist-test-slave-5kzz krb5kdc[14218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:44.766755 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:44.745757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41537 (local address 127.0.97.62:46597)
0620 23:24:44.747071 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:44.747077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:44.751438 (+ 4361us) server_negotiation.cc:407] Connection header received
0620 23:24:44.753264 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:44.753272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:44.753365 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:44.753483 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:44.755014 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:44.755745 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:44.757311 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:44.757712 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:44.757850 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:44.761300 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:44.761308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:44.761317 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:44.761349 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:44.763838 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:44.764522 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:44.764528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:44.764531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:44.764648 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:44.765064 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:44.765069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:44.765072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:44.765319 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:44.765474 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:44.766140 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 23:24:44.766362 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1166,"thread_start_us":1056,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:24:44.767765 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:44.746444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:41537)
0620 23:24:44.751259 (+ 4815us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:44.751318 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:44.752980 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:44.753676 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:44.753684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:44.754184 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:44.754839 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:44.754859 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:44.755923 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:44.755928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:44.757145 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:44.757157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:44.757964 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:44.757971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:44.758188 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:44.758998 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:44.759019 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:44.761100 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:44.763986 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:44.763995 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:44.764010 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:44.764403 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:44.764756 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:44.764760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:44.764764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:44.764948 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:44.765456 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:44.765464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:44.765781 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 23:24:44.766038 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 23:24:44.766538 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4535,"thread_start_us":4279,"threads_started":1}
I20260620 23:24:44.769256 14452 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:24:44.769549 14452 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:44.770201 14452 heartbeater.cc:511] Master 127.0.97.62:46597 requested a full tablet report, sending...
I20260620 23:24:44.772260 14263 ts_manager.cc:194] Registered new tserver with Master: a3b925c788004b839a529b99cdca8a3b (127.0.97.1:40321)
I20260620 23:24:44.773671 14263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:24:44.793646 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46597
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:24:45.106037 14458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:45.106298 14458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:45.106454 14458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:45.113626 14458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:45.113763 14458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:45.113945 14458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:24:45.123515 14458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:45.124826 14458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:45.126112 14458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:45.147115 14464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:45.147135 14463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:45.147404 14466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:45.147948 14458 server_base.cc:1034] running on GCE node
I20260620 23:24:45.148695 14458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:45.150014 14458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:45.151244 14458 hybrid_clock.cc:648] HybridClock initialized: now 1781997885151171 us; error 58 us; skew 500 ppm
I20260620 23:24:45.151659 14458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:45 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:45.155525 14458 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:24:45.157112 14458 webserver.cc:460] Webserver started at http://127.0.97.2:45635/ using document root <none> and password file <none>
I20260620 23:24:45.157960 14458 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:45.158095 14458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:45.158434 14458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:24:45.162204 14458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c"
format_stamp: "Formatted at 2026-06-20 23:24:45 on dist-test-slave-5kzz"
server_key: "8122c07c10b117dd30ac7a7a88d7e383"
server_key_iv: "5c73e3b6a97fe02f1b401afc52ad9cd8"
server_key_version: "encryptionkey@0"
I20260620 23:24:45.163204 14458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c"
format_stamp: "Formatted at 2026-06-20 23:24:45 on dist-test-slave-5kzz"
server_key: "8122c07c10b117dd30ac7a7a88d7e383"
server_key_iv: "5c73e3b6a97fe02f1b401afc52ad9cd8"
server_key_version: "encryptionkey@0"
I20260620 23:24:45.169560 14458 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260620 23:24:45.173843 14473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:45.175462 14458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:24:45.175611 14458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c"
format_stamp: "Formatted at 2026-06-20 23:24:45 on dist-test-slave-5kzz"
server_key: "8122c07c10b117dd30ac7a7a88d7e383"
server_key_iv: "5c73e3b6a97fe02f1b401afc52ad9cd8"
server_key_version: "encryptionkey@0"
I20260620 23:24:45.175817 14458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:45.201118 14458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:45.205404 14458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:45.205696 14458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:45.206627 14458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:45.207963 14458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:45.208041 14458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:45.208123 14458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:45.208168 14458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:45.238895 14458 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36637
I20260620 23:24:45.238976 14585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36637 every 8 connection(s)
I20260620 23:24:45.240151 14458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:24:45.240593 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 14458
I20260620 23:24:45.240774 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:24:45.241387 388 external_mini_cluster.cc:1428] Setting key ab08ea563a9b3df71a865050a2fdc9a9
Jun 20 23:24:45 dist-test-slave-5kzz krb5kdc[14218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:45.261389 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.242903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53905 (local address 127.0.97.62:46597)
0620 23:24:45.243100 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:45.243106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:45.244591 (+ 1485us) server_negotiation.cc:407] Connection header received
0620 23:24:45.246439 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:45.246444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:45.246536 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:45.246647 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:45.248312 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.249013 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.250550 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.251007 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.251148 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.255164 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:45.255169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:45.255173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:45.255204 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:45.257887 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:45.258890 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:45.258894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:45.258896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:45.259034 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:45.259626 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:45.259630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:45.259632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:45.259890 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:45.260002 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:45.260963 (+ 961us) server_negotiation.cc:299] Negotiation successful
0620 23:24:45.261144 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:24:45.263072 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.243625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:53905)
0620 23:24:45.244426 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:45.244467 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:45.246177 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:45.246900 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:45.246907 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:45.247569 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:45.248139 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.248156 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.249165 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.249177 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.250401 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.250412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.251231 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.251237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.251474 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.252167 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:45.252189 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:45.254912 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:45.258031 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:45.258043 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:45.258062 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:45.258772 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:45.259145 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:45.259153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:45.259160 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:45.259424 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:45.260031 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:45.260042 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:45.260538 (+ 496us) client_negotiation.cc:769] Sending connection context
0620 23:24:45.260883 (+ 345us) client_negotiation.cc:240] Negotiation successful
0620 23:24:45.261505 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":355,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:24:45.264626 14586 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46597
I20260620 23:24:45.265014 14586 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:45.265735 14586 heartbeater.cc:511] Master 127.0.97.62:46597 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 23:24:45.267313 14263 ts_manager.cc:194] Registered new tserver with Master: 28f99dbad7e7417b8b70e8f1dbc3d92c (127.0.97.2:36637)
I20260620 23:24:45.268126 14263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:24:45.293222 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46597
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:24:45.613000 14591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:45.613328 14591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:45.613498 14591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:45.620623 14591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:45.620776 14591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:45.620944 14591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:24:45.630350 14591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:45.631698 14591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:45.633121 14591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:45.646443 14599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:45.646512 14596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:45.646493 14597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:45.647255 14591 server_base.cc:1034] running on GCE node
I20260620 23:24:45.647836 14591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:45.648775 14591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:45.650002 14591 hybrid_clock.cc:648] HybridClock initialized: now 1781997885649958 us; error 56 us; skew 500 ppm
I20260620 23:24:45.650329 14591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:45 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:45.653550 14591 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:24:45.655084 14591 webserver.cc:460] Webserver started at http://127.0.97.3:46365/ using document root <none> and password file <none>
I20260620 23:24:45.655786 14591 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:45.655933 14591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:45.656251 14591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:24:45.659247 14591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "0ffc88719f5045b49a21c29f8dc2bffa"
format_stamp: "Formatted at 2026-06-20 23:24:45 on dist-test-slave-5kzz"
server_key: "5d2017dc36bb94dd12c2b1745cbd1819"
server_key_iv: "061a04182728ca123ad8ee85de838128"
server_key_version: "encryptionkey@0"
I20260620 23:24:45.660051 14591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0ffc88719f5045b49a21c29f8dc2bffa"
format_stamp: "Formatted at 2026-06-20 23:24:45 on dist-test-slave-5kzz"
server_key: "5d2017dc36bb94dd12c2b1745cbd1819"
server_key_iv: "061a04182728ca123ad8ee85de838128"
server_key_version: "encryptionkey@0"
I20260620 23:24:45.665318 14591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:24:45.668905 14606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:45.670258 14591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:24:45.670439 14591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "0ffc88719f5045b49a21c29f8dc2bffa"
format_stamp: "Formatted at 2026-06-20 23:24:45 on dist-test-slave-5kzz"
server_key: "5d2017dc36bb94dd12c2b1745cbd1819"
server_key_iv: "061a04182728ca123ad8ee85de838128"
server_key_version: "encryptionkey@0"
I20260620 23:24:45.670662 14591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:45.693322 14591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:45.697973 14591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:45.698280 14591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:45.699399 14591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:45.700666 14591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:45.700762 14591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:45.700899 14591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:45.700964 14591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:45.734267 14591 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39031
I20260620 23:24:45.734323 14718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39031 every 8 connection(s)
I20260620 23:24:45.735605 14591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:24:45.744589 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 14591
I20260620 23:24:45.744764 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:24:45.745289 388 external_mini_cluster.cc:1428] Setting key 770a3df61c91bef738e89b5e76973233
Jun 20 23:24:45 dist-test-slave-5kzz krb5kdc[14218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:45.755561 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.738272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56175 (local address 127.0.97.62:46597)
0620 23:24:45.738462 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:45.738467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:45.739599 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 23:24:45.741438 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:45.741445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:45.741535 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:45.741645 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:45.743188 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.743957 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.746204 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.746635 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.746809 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.750345 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:45.750350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:45.750353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:45.750385 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:45.753029 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:45.753657 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:45.753660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:45.753662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:45.753780 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:45.754131 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:45.754134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:45.754136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:45.754373 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:45.754477 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:45.755140 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 23:24:45.755318 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:24:45.756454 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.738756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:56175)
0620 23:24:45.739453 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:45.739489 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:45.741169 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:45.741899 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:45.741910 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:45.742425 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:45.743031 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.743047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.744116 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.744123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.746009 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.746021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.746912 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.746921 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.747203 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.747936 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:45.747957 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:45.750174 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:45.753155 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:45.753161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:45.753172 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:45.753557 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:45.753868 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:45.753871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:45.753874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:45.754047 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:45.754469 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:45.754474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:45.754801 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 23:24:45.755033 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 23:24:45.755420 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":270,"threads_started":1}
I20260620 23:24:45.757803 14719 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46597
I20260620 23:24:45.758060 14719 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:45.758775 14719 heartbeater.cc:511] Master 127.0.97.62:46597 requested a full tablet report, sending...
I20260620 23:24:45.760499 14262 ts_manager.cc:194] Registered new tserver with Master: 0ffc88719f5045b49a21c29f8dc2bffa (127.0.97.3:39031)
I20260620 23:24:45.761377 14262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56175
I20260620 23:24:45.772832 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:24:45.776477 14452 heartbeater.cc:503] Master 127.0.97.62:46597 was elected leader, sending a full tablet report...
I20260620 23:24:45.789876 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.777554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.0.97.62:46597)
0620 23:24:45.777751 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:45.777760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:45.778282 (+ 522us) server_negotiation.cc:407] Connection header received
0620 23:24:45.778443 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:45.778446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:45.778508 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:45.778592 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:45.780111 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.780905 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.782224 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.782610 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.782763 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.784927 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:45.784932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:45.784935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:45.784964 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:45.787548 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:45.788139 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:45.788142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:45.788144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:45.788251 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:45.788550 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:45.788553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:45.788554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:45.788827 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:45.789005 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:45.789242 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:24:45.789428 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:24:45.793767 14262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:24:45.800632 14262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:24:45.838413 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.812485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36637 (local address 127.0.0.1:37660)
0620 23:24:45.830215 (+ 17730us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:45.830232 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:45.830355 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:45.831299 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:45.831303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:45.831328 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:45.831740 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.831756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.833255 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.833263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.835721 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.835733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.837364 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.837370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.837578 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.837627 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:24:45.837736 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:24:45.837937 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17596,"mutex_wait_us":107,"thread_start_us":18099,"threads_started":1}
I20260620 23:24:45.839147 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.813465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.0.97.1:40321)
0620 23:24:45.829321 (+ 15856us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:45.829334 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:45.831175 (+ 1841us) server_negotiation.cc:407] Connection header received
0620 23:24:45.831276 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:45.831282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:45.831508 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:45.831712 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:45.832432 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.833771 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.836465 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.837457 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.837573 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.837807 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:45.838625 (+ 818us) server_negotiation.cc:299] Negotiation successful
0620 23:24:45.838832 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15628,"thread_start_us":15425,"threads_started":1}
I20260620 23:24:45.839144 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.813494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.0.97.2:36637)
0620 23:24:45.824391 (+ 10897us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:45.824404 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:45.830425 (+ 6021us) server_negotiation.cc:407] Connection header received
0620 23:24:45.830572 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:45.830584 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:45.830850 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:45.831043 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:45.831959 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.833121 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.835879 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.837167 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.837306 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.838599 (+ 1293us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:45.838661 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 23:24:45.838839 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10665,"thread_start_us":10496,"threads_started":1}
I20260620 23:24:45.839197 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.812225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39031 (local address 127.0.0.1:34544)
0620 23:24:45.828197 (+ 15972us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:45.828244 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:45.828465 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:45.830565 (+ 2100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:45.830576 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:45.830618 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:45.831150 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.831161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.833336 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.833340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.836188 (+ 2848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.836207 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.838188 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.838194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.838368 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.838402 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:24:45.838477 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:24:45.838811 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15722,"mutex_wait_us":163,"thread_start_us":15570,"threads_started":1}
I20260620 23:24:45.839496 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.812743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.0.97.3:39031)
0620 23:24:45.829840 (+ 17097us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:45.829848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:45.829905 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:24:45.830017 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:45.830025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:45.830317 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:45.831326 (+ 1009us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:45.831673 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.833218 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.836357 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.837687 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.837809 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.838045 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:45.838966 (+ 921us) server_negotiation.cc:299] Negotiation successful
0620 23:24:45.839120 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16941,"thread_start_us":16793,"threads_started":1}
I20260620 23:24:45.840687 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.812762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40321 (local address 127.0.0.1:43382)
0620 23:24:45.830903 (+ 18141us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:45.830919 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:45.831040 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:45.831804 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:45.831808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:45.831835 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:45.832250 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.832261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.833944 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.833948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.836320 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.836333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.837956 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.837962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.838138 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.838174 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:24:45.838261 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:24:45.838409 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18012,"mutex_wait_us":45,"thread_start_us":17184,"threads_started":1}
I20260620 23:24:45.843143 14654 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6cd18e180e40578ae073275fb275b6 (DEFAULT_TABLE table=test-table [id=aff248397f894bf6b9f6b07491ca0a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:24:45.843143 14521 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6cd18e180e40578ae073275fb275b6 (DEFAULT_TABLE table=test-table [id=aff248397f894bf6b9f6b07491ca0a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:24:45.844605 14654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6cd18e180e40578ae073275fb275b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:45.844607 14521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6cd18e180e40578ae073275fb275b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:45.844753 14387 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6cd18e180e40578ae073275fb275b6 (DEFAULT_TABLE table=test-table [id=aff248397f894bf6b9f6b07491ca0a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:24:45.845785 14387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6cd18e180e40578ae073275fb275b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:45.856544 14737 tablet_bootstrap.cc:492] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b: Bootstrap starting.
I20260620 23:24:45.858872 14739 tablet_bootstrap.cc:492] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa: Bootstrap starting.
I20260620 23:24:45.860630 14738 tablet_bootstrap.cc:492] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c: Bootstrap starting.
I20260620 23:24:45.860673 14737 tablet_bootstrap.cc:654] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:45.862087 14737 log.cc:824] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:45.864061 14738 tablet_bootstrap.cc:654] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:45.864826 14739 tablet_bootstrap.cc:654] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:45.865247 14738 log.cc:824] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:45.865360 14737 tablet_bootstrap.cc:492] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b: No bootstrap required, opened a new log
I20260620 23:24:45.865621 14737 ts_tablet_manager.cc:1397] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260620 23:24:45.865880 14739 log.cc:824] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:45.868144 14738 tablet_bootstrap.cc:492] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c: No bootstrap required, opened a new log
I20260620 23:24:45.868377 14738 ts_tablet_manager.cc:1397] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260620 23:24:45.870134 14739 tablet_bootstrap.cc:492] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa: No bootstrap required, opened a new log
I20260620 23:24:45.870412 14739 ts_tablet_manager.cc:1397] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260620 23:24:45.875999 14737 raft_consensus.cc:348] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:45.876437 14737 raft_consensus.cc:374] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:45.876533 14737 raft_consensus.cc:729] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b925c788004b839a529b99cdca8a3b, State: Initialized, Role: FOLLOWER
I20260620 23:24:45.877154 14737 consensus_queue.cc:260] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:45.878325 14739 raft_consensus.cc:348] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:45.878780 14739 raft_consensus.cc:374] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:45.878875 14739 raft_consensus.cc:729] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ffc88719f5045b49a21c29f8dc2bffa, State: Initialized, Role: FOLLOWER
I20260620 23:24:45.878623 14738 raft_consensus.cc:348] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:45.879035 14737 ts_tablet_manager.cc:1428] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260620 23:24:45.879058 14738 raft_consensus.cc:374] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:45.879138 14738 raft_consensus.cc:729] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f99dbad7e7417b8b70e8f1dbc3d92c, State: Initialized, Role: FOLLOWER
I20260620 23:24:45.879571 14739 consensus_queue.cc:260] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:45.879868 14738 consensus_queue.cc:260] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:45.883500 14586 heartbeater.cc:503] Master 127.0.97.62:46597 was elected leader, sending a full tablet report...
I20260620 23:24:45.884790 14738 ts_tablet_manager.cc:1428] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c: Time spent starting tablet: real 0.016s user 0.005s sys 0.009s
I20260620 23:24:45.885690 14719 heartbeater.cc:503] Master 127.0.97.62:46597 was elected leader, sending a full tablet report...
I20260620 23:24:45.886380 14739 ts_tablet_manager.cc:1428] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260620 23:24:45.935928 14743 raft_consensus.cc:482] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:24:45.936329 14743 raft_consensus.cc:504] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:45.938040 14743 leader_election.cc:288] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ffc88719f5045b49a21c29f8dc2bffa (127.0.97.3:39031), 28f99dbad7e7417b8b70e8f1dbc3d92c (127.0.97.2:36637)
I20260620 23:24:45.945747 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.938436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46761 (local address 127.0.97.3:39031)
0620 23:24:45.938632 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:45.938640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:45.939513 (+ 873us) server_negotiation.cc:407] Connection header received
0620 23:24:45.939742 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:45.939747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:45.939804 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:45.939907 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:45.940446 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.941157 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.943982 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.944798 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.944954 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.945059 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:45.945355 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 23:24:45.945479 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:24:45.945895 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.938442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39031 (local address 127.0.97.1:46761)
0620 23:24:45.939391 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:45.939408 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:45.939592 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:45.939936 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:45.939940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:45.939963 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:45.940297 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.940307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.941284 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.941291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.943826 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.943842 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.945005 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.945011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.945189 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.945217 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:24:45.945328 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 23:24:45.945462 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":103,"thread_start_us":607,"threads_started":1}
I20260620 23:24:45.945936 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.938499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36363 (local address 127.0.97.2:36637)
0620 23:24:45.938732 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:45.938736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:45.939695 (+ 959us) server_negotiation.cc:407] Connection header received
0620 23:24:45.939889 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:45.939893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:45.939945 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:45.940038 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:45.940631 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.941405 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.944033 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.945011 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.945168 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.945269 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:45.945552 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 23:24:45.945678 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:24:45.946231 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.938841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36637 (local address 127.0.97.1:36363)
0620 23:24:45.939590 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:45.939604 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:45.939748 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:45.940069 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:45.940073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:45.940112 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:45.940485 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.940492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.941528 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.941534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.943841 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:45.943859 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.945235 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.945242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:45.945397 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.945419 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:24:45.945525 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:24:45.945653 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":83,"thread_start_us":429,"threads_started":1}
I20260620 23:24:45.947021 14674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6cd18e180e40578ae073275fb275b6" candidate_uuid: "a3b925c788004b839a529b99cdca8a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" is_pre_election: true
I20260620 23:24:45.947233 14541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6cd18e180e40578ae073275fb275b6" candidate_uuid: "a3b925c788004b839a529b99cdca8a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" is_pre_election: true
I20260620 23:24:45.947654 14674 raft_consensus.cc:2454] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3b925c788004b839a529b99cdca8a3b in term 0.
I20260620 23:24:45.947732 14541 raft_consensus.cc:2454] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3b925c788004b839a529b99cdca8a3b in term 0.
I20260620 23:24:45.948436 14340 leader_election.cc:302] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ffc88719f5045b49a21c29f8dc2bffa, a3b925c788004b839a529b99cdca8a3b; no voters:
I20260620 23:24:45.948843 14743 raft_consensus.cc:2781] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:24:45.948988 14743 raft_consensus.cc:482] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:24:45.949092 14743 raft_consensus.cc:3037] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:45.951357 14743 raft_consensus.cc:504] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:45.951985 14743 leader_election.cc:288] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [CANDIDATE]: Term 1 election: Requested vote from peers 0ffc88719f5045b49a21c29f8dc2bffa (127.0.97.3:39031), 28f99dbad7e7417b8b70e8f1dbc3d92c (127.0.97.2:36637)
I20260620 23:24:45.952443 14541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6cd18e180e40578ae073275fb275b6" candidate_uuid: "a3b925c788004b839a529b99cdca8a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c"
I20260620 23:24:45.952467 14674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6cd18e180e40578ae073275fb275b6" candidate_uuid: "a3b925c788004b839a529b99cdca8a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ffc88719f5045b49a21c29f8dc2bffa"
I20260620 23:24:45.952665 14541 raft_consensus.cc:3037] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:45.952687 14674 raft_consensus.cc:3037] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:45.955103 14541 raft_consensus.cc:2454] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3b925c788004b839a529b99cdca8a3b in term 1.
I20260620 23:24:45.955129 14674 raft_consensus.cc:2454] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3b925c788004b839a529b99cdca8a3b in term 1.
I20260620 23:24:45.955503 14341 leader_election.cc:302] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28f99dbad7e7417b8b70e8f1dbc3d92c, a3b925c788004b839a529b99cdca8a3b; no voters:
I20260620 23:24:45.955796 14743 raft_consensus.cc:2781] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:24:45.956173 14743 raft_consensus.cc:686] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 1 LEADER]: Becoming Leader. State: Replica: a3b925c788004b839a529b99cdca8a3b, State: Running, Role: LEADER
I20260620 23:24:45.956619 14743 consensus_queue.cc:237] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:45.961272 14262 catalog_manager.cc:5495] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to a3b925c788004b839a529b99cdca8a3b (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a3b925c788004b839a529b99cdca8a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } health_report { overall_health: HEALTHY } } }
W20260620 23:24:45.987358 14720 tablet.cc:2369] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:24:45.992336 14587 tablet.cc:2369] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:24:45.995143 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:45.989741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.0.97.1:40321)
0620 23:24:45.989955 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:45.989962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:45.989983 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:24:45.990092 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:45.990096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:45.990168 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:45.990330 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:24:45.990666 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.991414 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.992904 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:45.993320 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:45.993468 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:45.994014 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:24:45.994543 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:24:45.994741 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:45.994792 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 23:24:45.994873 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 23:24:45.995428 14453 tablet.cc:2369] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:24:46.003170 14541 raft_consensus.cc:1261] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 1 FOLLOWER]: Refusing update from remote peer a3b925c788004b839a529b99cdca8a3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:24:46.003218 14674 raft_consensus.cc:1261] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa [term 1 FOLLOWER]: Refusing update from remote peer a3b925c788004b839a529b99cdca8a3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:24:46.004215 14748 consensus_queue.cc:1035] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:24:46.004554 14743 consensus_queue.cc:1035] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:24:46.015398 14751 mvcc.cc:205] Tried to move back new op lower bound from 7299063341059117056 to 7299063340881825792. Current Snapshot: MvccSnapshot[applied={T|T < 7299063341059117056}]
I20260620 23:24:46.017464 14753 mvcc.cc:205] Tried to move back new op lower bound from 7299063341059117056 to 7299063340881825792. Current Snapshot: MvccSnapshot[applied={T|T < 7299063341059117056}]
I20260620 23:24:46.018988 14752 mvcc.cc:205] Tried to move back new op lower bound from 7299063341059117056 to 7299063340881825792. Current Snapshot: MvccSnapshot[applied={T|T < 7299063341059117056}]
I20260620 23:24:46.030763 14262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:24:46.031055 14262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:24:46.034535 14262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Sending DeleteTablet for 3 replicas of tablet 8c6cd18e180e40578ae073275fb275b6
I20260620 23:24:46.035754 14654 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6cd18e180e40578ae073275fb275b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:24:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34544
I20260620 23:24:46.035754 14521 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6cd18e180e40578ae073275fb275b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:24:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37660
I20260620 23:24:46.035804 14387 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6cd18e180e40578ae073275fb275b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:24:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43382
I20260620 23:24:46.038450 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 14324
I20260620 23:24:46.053090 14763 tablet_replica.cc:330] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b: stopping tablet replica
I20260620 23:24:46.053591 14763 raft_consensus.cc:2229] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:24:46.054275 14763 raft_consensus.cc:2258] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:24:46.054307 14761 tablet_replica.cc:330] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c: stopping tablet replica
I20260620 23:24:46.054569 14762 tablet_replica.cc:330] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa: stopping tablet replica
I20260620 23:24:46.055066 14762 raft_consensus.cc:2229] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:24:46.055583 14761 raft_consensus.cc:2229] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:24:46.055682 14762 raft_consensus.cc:2258] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:24:46.056653 14763 ts_tablet_manager.cc:1905] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:24:46.057160 14761 raft_consensus.cc:2258] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:24:46.057853 14762 ts_tablet_manager.cc:1905] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:24:46.059216 14761 ts_tablet_manager.cc:1905] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:24:46.064165 14763 ts_tablet_manager.cc:1918] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:24:46.064327 14763 log.cc:1196] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/8c6cd18e180e40578ae073275fb275b6
I20260620 23:24:46.064851 14763 ts_tablet_manager.cc:1939] T 8c6cd18e180e40578ae073275fb275b6 P a3b925c788004b839a529b99cdca8a3b: Deleting consensus metadata
I20260620 23:24:46.065119 14762 ts_tablet_manager.cc:1918] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:24:46.065308 14762 log.cc:1196] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/8c6cd18e180e40578ae073275fb275b6
I20260620 23:24:46.065930 14762 ts_tablet_manager.cc:1939] T 8c6cd18e180e40578ae073275fb275b6 P 0ffc88719f5045b49a21c29f8dc2bffa: Deleting consensus metadata
I20260620 23:24:46.067026 14250 catalog_manager.cc:4841] TS a3b925c788004b839a529b99cdca8a3b (127.0.97.1:40321): tablet 8c6cd18e180e40578ae073275fb275b6 (table test-table [id=aff248397f894bf6b9f6b07491ca0a37]) successfully deleted
I20260620 23:24:46.067754 14247 catalog_manager.cc:4841] TS 0ffc88719f5045b49a21c29f8dc2bffa (127.0.97.3:39031): tablet 8c6cd18e180e40578ae073275fb275b6 (table test-table [id=aff248397f894bf6b9f6b07491ca0a37]) successfully deleted
Jun 20 23:24:46 dist-test-slave-5kzz krb5kdc[14218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:24:46.071002 14761 ts_tablet_manager.cc:1918] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:24:46.071166 14761 log.cc:1196] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/8c6cd18e180e40578ae073275fb275b6
I20260620 23:24:46.071717 14761 ts_tablet_manager.cc:1939] T 8c6cd18e180e40578ae073275fb275b6 P 28f99dbad7e7417b8b70e8f1dbc3d92c: Deleting consensus metadata
I20260620 23:24:46.073127 14248 catalog_manager.cc:4841] TS 28f99dbad7e7417b8b70e8f1dbc3d92c (127.0.97.2:36637): tablet 8c6cd18e180e40578ae073275fb275b6 (table test-table [id=aff248397f894bf6b9f6b07491ca0a37]) successfully deleted
I20260620 23:24:46.076637 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:46.039546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.0.97.1:40321)
0620 23:24:46.039724 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:46.039730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:46.045236 (+ 5506us) server_negotiation.cc:407] Connection header received
0620 23:24:46.046325 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:46.046333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:46.046464 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:46.046582 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:46.051334 (+ 4752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:46.052987 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:46.059664 (+ 6677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:46.060412 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:46.060508 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:46.069498 (+ 8990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:46.069512 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:46.069531 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:46.069575 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:46.073548 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:46.074285 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:46.074294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:46.074298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:46.074494 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:46.074943 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:46.074949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:46.074952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:46.075395 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:46.075744 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:46.076033 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 23:24:46.076266 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:24:46.077268 14447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:46.176522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 14324
I20260620 23:24:46.196403 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 14458
Jun 20 23:24:46 dist-test-slave-5kzz krb5kdc[14218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:24:46.211225 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:46.197097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.0.97.2:36637)
0620 23:24:46.197291 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:46.197295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:46.197313 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:24:46.197492 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:46.197497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:46.197572 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:46.197665 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:46.199068 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:46.199782 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:46.200987 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:46.201358 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:46.201453 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:46.205747 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:46.205762 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:46.205777 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:46.205811 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:46.208649 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:46.209365 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:46.209381 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:46.209384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:46.209519 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:46.209822 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:46.209825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:46.209827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:46.210189 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:46.210412 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:46.210617 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 23:24:46.210865 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:24:46.211872 14581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:46.297636 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 14458
I20260620 23:24:46.314154 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 14591
Jun 20 23:24:46 dist-test-slave-5kzz krb5kdc[14218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:24:46.329015 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:46.314964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34552 (local address 127.0.97.3:39031)
0620 23:24:46.315199 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:46.315205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:46.315227 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:24:46.315454 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:46.315458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:46.315525 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:46.315637 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:46.317030 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:46.317761 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:46.319232 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:46.319632 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:46.319727 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:46.323955 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:46.323970 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:46.323986 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:46.324026 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:46.326793 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:46.327349 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:46.327383 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:46.327385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:46.327526 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:46.327814 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:46.327817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:46.327818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:46.328143 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:46.328393 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:46.328611 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:24:46.328770 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:24:46.329663 14714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:46.417860 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 14591
I20260620 23:24:46.434600 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 14231
I20260620 23:24:46.435786 14293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:46.545465 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 14231
I20260620 23:24:46.567340 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46597
--webserver_interface=127.0.97.62
--webserver_port=42747
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:24:46.880824 14769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:46.881088 14769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:46.881191 14769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:46.888121 14769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:24:46.888211 14769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:46.888253 14769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:46.888309 14769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:24:46.888345 14769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:24:46.897485 14769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:46.898627 14769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:46.899852 14769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:46.920576 14775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:46.920609 14778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:46.920631 14776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:46.921425 14769 server_base.cc:1034] running on GCE node
I20260620 23:24:46.922246 14769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:46.923754 14769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:46.924958 14769 hybrid_clock.cc:648] HybridClock initialized: now 1781997886924915 us; error 31 us; skew 500 ppm
I20260620 23:24:46.925366 14769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:46 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:46.928817 14769 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:24:46.930274 14769 webserver.cc:460] Webserver started at http://127.0.97.62:42747/ using document root <none> and password file <none>
I20260620 23:24:46.931003 14769 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:46.931134 14769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:46.936710 14769 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:24:46.939491 14785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:46.940786 14769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:24:46.940972 14769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "2596abde319b46c7a2b06674a8c3f2c9"
format_stamp: "Formatted at 2026-06-20 23:24:44 on dist-test-slave-5kzz"
server_key: "368bb5da9634f96db0e56532ddbdbf65"
server_key_iv: "f65942ebe855af550b48199d526a23b4"
server_key_version: "encryptionkey@0"
I20260620 23:24:46.941992 14769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:46.960556 14769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:46.964639 14769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:46.964913 14769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:46.983505 14769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46597
I20260620 23:24:46.983525 14836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46597 every 8 connection(s)
I20260620 23:24:46.985037 14769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:24:46.988533 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 14769
I20260620 23:24:46.989245 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40321
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46597
--webserver_port=36769
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:24:46.992226 14837 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:24:46.996483 14837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Bootstrap starting.
I20260620 23:24:47.016121 14837 log.cc:824] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:47.028236 14837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:24:47.028779 14837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Bootstrap complete.
I20260620 23:24:47.037796 14837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } }
I20260620 23:24:47.038584 14837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2596abde319b46c7a2b06674a8c3f2c9, State: Initialized, Role: FOLLOWER
I20260620 23:24:47.039261 14837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } }
I20260620 23:24:47.039498 14837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:24:47.039613 14837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:24:47.039799 14837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:24:47.043646 14837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } }
I20260620 23:24:47.044191 14837 leader_election.cc:302] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2596abde319b46c7a2b06674a8c3f2c9; no voters:
I20260620 23:24:47.044847 14837 leader_election.cc:288] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:24:47.044999 14841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:24:47.046131 14841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 2 LEADER]: Becoming Leader. State: Replica: 2596abde319b46c7a2b06674a8c3f2c9, State: Running, Role: LEADER
I20260620 23:24:47.046758 14841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } }
I20260620 23:24:47.047273 14837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:24:47.049258 14842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2596abde319b46c7a2b06674a8c3f2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } } }
I20260620 23:24:47.049674 14842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:47.051314 14843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2596abde319b46c7a2b06674a8c3f2c9. Latest consensus state: current_term: 2 leader_uuid: "2596abde319b46c7a2b06674a8c3f2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } } }
I20260620 23:24:47.051564 14846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:24:47.051664 14843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:47.060698 14846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:24:47.062145 14846 catalog_manager.cc:1259] Loaded cluster ID: 3401ee2860bd4a168eeec4ab5b478eff
I20260620 23:24:47.062314 14846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:24:47.065438 14846 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 23:24:47.083451 14846 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 23:24:47.102928 14846 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:24:47.105121 14846 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Loaded TSK: 0
I20260620 23:24:47.106611 14846 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:24:47.331912 14839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:47.332219 14839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:47.332412 14839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:47.339437 14839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:47.339563 14839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:47.339711 14839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:24:47.349666 14839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36769
--tserver_master_addrs=127.0.97.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:47.350956 14839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:47.352277 14839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:47.366925 14865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:47.371786 14867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:47.372844 14839 server_base.cc:1034] running on GCE node
W20260620 23:24:47.374346 14864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:47.375124 14839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:47.376156 14839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:47.377373 14839 hybrid_clock.cc:648] HybridClock initialized: now 1781997887377300 us; error 66 us; skew 500 ppm
I20260620 23:24:47.377681 14839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:47 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:47.380739 14839 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:24:47.383975 14839 webserver.cc:460] Webserver started at http://127.0.97.1:36769/ using document root <none> and password file <none>
I20260620 23:24:47.384707 14839 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:47.384871 14839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:47.390295 14839 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:24:47.393188 14874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:47.394618 14839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:24:47.394850 14839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "a3b925c788004b839a529b99cdca8a3b"
format_stamp: "Formatted at 2026-06-20 23:24:44 on dist-test-slave-5kzz"
server_key: "ae8c8aee91e618a594e0cd31b5f9be94"
server_key_iv: "231f70efbe16e01bc54f5c30f201f54d"
server_key_version: "encryptionkey@0"
I20260620 23:24:47.395776 14839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:47.418021 14839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:47.422354 14839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:47.422631 14839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:47.423764 14839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:47.425158 14839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:47.425244 14839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:47.425400 14839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:47.425496 14839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:47.460819 14839 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40321
I20260620 23:24:47.460908 14986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40321 every 8 connection(s)
I20260620 23:24:47.462188 14839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:24:47.472090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 14839
I20260620 23:24:47.472748 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36637
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46597
--webserver_port=45635
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:24:47 dist-test-slave-5kzz krb5kdc[14218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:47.486954 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:47.465434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:59071)
0620 23:24:47.466289 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:47.466333 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:47.467995 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:47.469359 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:47.469371 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:47.470165 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:47.470983 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:47.471002 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:47.472664 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:47.472670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:47.473936 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:47.473948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:47.475378 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:47.475386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:47.475656 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:47.476362 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:47.476397 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:47.479050 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:47.482793 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:47.482802 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:47.482816 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:47.483218 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:47.483668 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:47.483673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:47.483677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:47.483856 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:47.484572 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:47.484579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:47.484897 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 23:24:47.485148 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 23:24:47.485662 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":365,"threads_started":1}
I20260620 23:24:47.486951 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:47.465162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59071 (local address 127.0.97.62:46597)
0620 23:24:47.465681 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:47.465692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:47.466468 (+ 776us) server_negotiation.cc:407] Connection header received
0620 23:24:47.468543 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:47.468555 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:47.468869 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:47.469225 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:47.471203 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:47.472510 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:47.474077 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:47.474881 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:47.475050 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:47.479333 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:47.479346 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:47.479369 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:47.479421 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:47.482609 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:47.483364 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:47.483373 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:47.483380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:47.483549 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:47.483976 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:47.483981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:47.483984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:47.484424 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:47.484625 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:47.485358 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 23:24:47.485716 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":223,"threads_started":1}
I20260620 23:24:47.490800 14987 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46597
I20260620 23:24:47.491138 14987 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:47.491814 14987 heartbeater.cc:511] Master 127.0.97.62:46597 requested a full tablet report, sending...
I20260620 23:24:47.494313 14802 ts_manager.cc:194] Registered new tserver with Master: a3b925c788004b839a529b99cdca8a3b (127.0.97.1:40321)
I20260620 23:24:47.496727 14802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59071
W20260620 23:24:47.788165 14991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:47.788486 14991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:47.788575 14991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:47.795300 14991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:47.795387 14991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:47.795497 14991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:24:47.804522 14991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45635
--tserver_master_addrs=127.0.97.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:47.805627 14991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:47.806772 14991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:47.819177 14996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:47.819132 14997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:47.819214 14999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:47.820071 14991 server_base.cc:1034] running on GCE node
I20260620 23:24:47.820704 14991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:47.821580 14991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:47.822774 14991 hybrid_clock.cc:648] HybridClock initialized: now 1781997887822734 us; error 52 us; skew 500 ppm
I20260620 23:24:47.823079 14991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:47 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:47.826135 14991 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:24:47.827455 14991 webserver.cc:460] Webserver started at http://127.0.97.2:45635/ using document root <none> and password file <none>
I20260620 23:24:47.828056 14991 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:47.828159 14991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:47.833057 14991 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 23:24:47.835680 15006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:47.836934 14991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:24:47.837069 14991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c"
format_stamp: "Formatted at 2026-06-20 23:24:45 on dist-test-slave-5kzz"
server_key: "8122c07c10b117dd30ac7a7a88d7e383"
server_key_iv: "5c73e3b6a97fe02f1b401afc52ad9cd8"
server_key_version: "encryptionkey@0"
I20260620 23:24:47.837946 14991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:47.851109 14991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:47.855264 14991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:47.855525 14991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:47.856395 14991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:47.857605 14991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:47.857720 14991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:47.857839 14991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:47.857921 14991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:47.887899 14991 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36637
I20260620 23:24:47.887950 15118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36637 every 8 connection(s)
I20260620 23:24:47.889295 14991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:24:47.895035 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 14991
I20260620 23:24:47.895637 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39031
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46597
--webserver_port=46365
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:24:47 dist-test-slave-5kzz krb5kdc[14218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:47.909364 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:47.891769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54917 (local address 127.0.97.62:46597)
0620 23:24:47.891995 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:47.892000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:47.893162 (+ 1162us) server_negotiation.cc:407] Connection header received
0620 23:24:47.894921 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:47.894926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:47.895016 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:47.895147 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:47.896737 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:47.897430 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:47.899057 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:47.899527 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:47.899690 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:47.903591 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:47.903596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:47.903600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:47.903630 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:47.906048 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:47.907059 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:47.907065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:47.907071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:47.907197 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:47.907773 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:47.907776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:47.907778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:47.908026 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:47.908135 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:47.908975 (+ 840us) server_negotiation.cc:299] Negotiation successful
0620 23:24:47.909136 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:24:47.910876 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:47.892235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:54917)
0620 23:24:47.892952 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:47.893026 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:47.894610 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:47.895380 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:47.895388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:47.895998 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:47.896566 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:47.896584 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:47.897583 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:47.897589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:47.898914 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:47.898927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:47.899770 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:47.899777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:47.900001 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:47.900730 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:47.900751 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:47.903387 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:47.906219 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:47.906230 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:47.906248 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:47.906901 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:47.907329 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:47.907336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:47.907342 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:47.907656 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:47.908156 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:47.908166 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:47.908590 (+ 424us) client_negotiation.cc:769] Sending connection context
0620 23:24:47.908888 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 23:24:47.909428 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":329,"threads_started":1}
I20260620 23:24:47.912309 15119 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46597
I20260620 23:24:47.912662 15119 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:47.913601 15119 heartbeater.cc:511] Master 127.0.97.62:46597 requested a full tablet report, sending...
I20260620 23:24:47.915217 14802 ts_manager.cc:194] Registered new tserver with Master: 28f99dbad7e7417b8b70e8f1dbc3d92c (127.0.97.2:36637)
I20260620 23:24:47.916007 14802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54917
W20260620 23:24:48.224850 15122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:48.225198 15122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:48.225368 15122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:48.232465 15122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:48.232605 15122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:48.232788 15122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:24:48.242508 15122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46365
--tserver_master_addrs=127.0.97.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:48.243803 15122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:48.245114 15122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:48.258644 15127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:48.258723 15130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:48.258627 15128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:48.259531 15122 server_base.cc:1034] running on GCE node
I20260620 23:24:48.260120 15122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:48.260988 15122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:48.262207 15122 hybrid_clock.cc:648] HybridClock initialized: now 1781997888262183 us; error 67 us; skew 500 ppm
I20260620 23:24:48.262508 15122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:48 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:48.274356 15122 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:24:48.276000 15122 webserver.cc:460] Webserver started at http://127.0.97.3:46365/ using document root <none> and password file <none>
I20260620 23:24:48.276664 15122 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:48.276775 15122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:48.281811 15122 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 23:24:48.284834 15137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:48.286095 15122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:24:48.286226 15122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "0ffc88719f5045b49a21c29f8dc2bffa"
format_stamp: "Formatted at 2026-06-20 23:24:45 on dist-test-slave-5kzz"
server_key: "5d2017dc36bb94dd12c2b1745cbd1819"
server_key_iv: "061a04182728ca123ad8ee85de838128"
server_key_version: "encryptionkey@0"
I20260620 23:24:48.287226 15122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:48.312603 15122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:48.316857 15122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:48.317085 15122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:48.318181 15122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:48.319482 15122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:48.319605 15122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:48.319676 15122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:48.319731 15122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:48.352926 15122 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39031
I20260620 23:24:48.352982 15249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39031 every 8 connection(s)
I20260620 23:24:48.354398 15122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:24:48.359810 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 15122
Jun 20 23:24:48 dist-test-slave-5kzz krb5kdc[14218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:24:48.377121 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.356926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47929 (local address 127.0.97.62:46597)
0620 23:24:48.357187 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:48.357203 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:48.358291 (+ 1088us) server_negotiation.cc:407] Connection header received
0620 23:24:48.360609 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:48.360615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:48.360705 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:48.360823 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:48.362435 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.363443 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.365856 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.366522 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.366648 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.370385 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:48.370390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:48.370393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:48.370422 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:48.373924 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:48.374669 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:48.374704 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:48.374710 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:48.374839 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:48.375282 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:48.375288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:48.375292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:48.375646 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:48.375766 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:48.376544 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 23:24:48.376754 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:24:48.378228 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.357405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:47929)
0620 23:24:48.358126 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:48.358164 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:48.360313 (+ 2149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:48.361069 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:48.361077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:48.361731 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:24:48.362267 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.362283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.363639 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.363643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.365706 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.365718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.366772 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.366778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.367073 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.367768 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:24:48.367798 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:24:48.370170 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:24:48.374068 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:48.374079 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:48.374099 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:48.374553 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:48.374959 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:24:48.374962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:24:48.374965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:24:48.375170 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:24:48.375762 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:24:48.375771 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:24:48.376161 (+ 390us) client_negotiation.cc:769] Sending connection context
0620 23:24:48.376431 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 23:24:48.376952 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":273,"threads_started":1}
I20260620 23:24:48.378245 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.360737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.0.97.62:46597)
0620 23:24:48.363079 (+ 2342us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:48.363085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:48.363433 (+ 348us) server_negotiation.cc:407] Connection header received
0620 23:24:48.363563 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:48.363568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:48.363639 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:48.363713 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:48.365627 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.366773 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.368589 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.369044 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.369215 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.372138 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:48.372147 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:48.372152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:48.372205 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:48.375515 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:48.376239 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:48.376246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:48.376249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:48.376379 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:48.376746 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:48.376752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:48.376755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:48.377039 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:48.377300 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:48.377689 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 23:24:48.377899 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2229,"thread_start_us":2096,"threads_started":1}
I20260620 23:24:48.379755 15250 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46597
I20260620 23:24:48.380054 15250 heartbeater.cc:464] Registering TS with master...
I20260620 23:24:48.380821 15250 heartbeater.cc:511] Master 127.0.97.62:46597 requested a full tablet report, sending...
I20260620 23:24:48.382131 14802 ts_manager.cc:194] Registered new tserver with Master: 0ffc88719f5045b49a21c29f8dc2bffa (127.0.97.3:39031)
I20260620 23:24:48.383023 14802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47929
I20260620 23:24:48.392832 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:24:48.399508 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.393792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52432 (local address 127.0.97.62:46597)
0620 23:24:48.394044 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:48.394050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:48.394273 (+ 223us) server_negotiation.cc:407] Connection header received
0620 23:24:48.394462 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:48.394466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:48.394598 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:48.394779 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:24:48.395178 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.395847 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.397300 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.397729 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.397832 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.398288 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:24:48.398938 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:24:48.399049 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:48.399147 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 23:24:48.399212 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:24:48.400406 14802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:24:48.407366 14802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:24:48.428180 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.418927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.0.97.3:39031)
0620 23:24:48.420163 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:48.420175 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:48.420218 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:24:48.420346 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:48.420358 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:48.420678 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:48.420910 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:48.421760 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.423262 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.425979 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.426953 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.427102 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.427348 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:48.427673 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:24:48.427834 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":909,"threads_started":1}
I20260620 23:24:48.428352 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.418269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39031 (local address 127.0.0.1:34566)
0620 23:24:48.419224 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:48.419277 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:48.419476 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:48.420952 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:48.420963 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:48.421009 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:48.421513 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.421526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.423415 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.423421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.425799 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.425809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.427157 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.427163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.427438 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.427482 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 23:24:48.427646 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 23:24:48.427860 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":100,"thread_start_us":497,"threads_started":1}
I20260620 23:24:48.430876 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.418370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36637 (local address 127.0.0.1:37690)
0620 23:24:48.421765 (+ 3395us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:48.421787 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:48.422029 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:48.422832 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:48.422836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:48.422862 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:48.423279 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.423291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.425299 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.425306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.427512 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.427527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.430012 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.430018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.430199 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.430221 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:24:48.430338 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:24:48.430464 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3225,"mutex_wait_us":29,"thread_start_us":4844,"threads_started":1}
I20260620 23:24:48.430918 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.418742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.0.97.2:36637)
0620 23:24:48.419973 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:48.419982 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:48.421899 (+ 1917us) server_negotiation.cc:407] Connection header received
0620 23:24:48.422250 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:48.422262 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:48.422531 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:48.422760 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:48.423491 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.425017 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.427691 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.429779 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.429932 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.430130 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:48.430396 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:24:48.430575 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":1532,"threads_started":1}
I20260620 23:24:48.432250 15185 tablet_service.cc:1467] Processing CreateTablet for tablet 7114a1b70926407da8d3295bdd7424ff (DEFAULT_TABLE table=test-table [id=b3a6bde75ab0477ea9644d8f3213961c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:24:48.433386 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.419437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40321 (local address 127.0.0.1:43420)
0620 23:24:48.423487 (+ 4050us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:48.423501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:48.423653 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:48.424146 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:48.424152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:48.424175 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:48.425262 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.425275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.427227 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.427232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.429371 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.429383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.432555 (+ 3172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.432570 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.432745 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.432776 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:24:48.432881 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:24:48.433016 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3923,"mutex_wait_us":3,"thread_start_us":1958,"threads_started":1}
I20260620 23:24:48.433882 15185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7114a1b70926407da8d3295bdd7424ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:48.434060 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.419974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43420 (local address 127.0.97.1:40321)
0620 23:24:48.420506 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:48.420513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:48.423612 (+ 3099us) server_negotiation.cc:407] Connection header received
0620 23:24:48.423817 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:48.423823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:48.423969 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:48.424142 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:48.425492 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.427074 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.430816 (+ 3742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.432329 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.432472 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.432647 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:48.433616 (+ 969us) server_negotiation.cc:299] Negotiation successful
0620 23:24:48.433792 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":474,"threads_started":1}
I20260620 23:24:48.434541 15054 tablet_service.cc:1467] Processing CreateTablet for tablet 7114a1b70926407da8d3295bdd7424ff (DEFAULT_TABLE table=test-table [id=b3a6bde75ab0477ea9644d8f3213961c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:24:48.435827 15054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7114a1b70926407da8d3295bdd7424ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:48.437489 14922 tablet_service.cc:1467] Processing CreateTablet for tablet 7114a1b70926407da8d3295bdd7424ff (DEFAULT_TABLE table=test-table [id=b3a6bde75ab0477ea9644d8f3213961c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:24:48.438845 14922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7114a1b70926407da8d3295bdd7424ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:24:48.446668 15266 tablet_bootstrap.cc:492] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa: Bootstrap starting.
I20260620 23:24:48.449627 15266 tablet_bootstrap.cc:654] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:48.450802 15266 log.cc:824] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:48.454154 15266 tablet_bootstrap.cc:492] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa: No bootstrap required, opened a new log
I20260620 23:24:48.454360 15269 tablet_bootstrap.cc:492] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b: Bootstrap starting.
I20260620 23:24:48.454525 15266 ts_tablet_manager.cc:1397] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 23:24:48.456982 15267 tablet_bootstrap.cc:492] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c: Bootstrap starting.
I20260620 23:24:48.458151 15269 tablet_bootstrap.cc:654] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:48.459957 15269 log.cc:824] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:48.460353 15267 tablet_bootstrap.cc:654] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c: Neither blocks nor log segments found. Creating new log.
I20260620 23:24:48.461635 15267 log.cc:824] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:48.467526 15266 raft_consensus.cc:348] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:48.468204 15266 raft_consensus.cc:374] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:48.468376 15266 raft_consensus.cc:729] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ffc88719f5045b49a21c29f8dc2bffa, State: Initialized, Role: FOLLOWER
I20260620 23:24:48.469678 15266 consensus_queue.cc:260] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:48.472157 15250 heartbeater.cc:503] Master 127.0.97.62:46597 was elected leader, sending a full tablet report...
I20260620 23:24:48.473169 15269 tablet_bootstrap.cc:492] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b: No bootstrap required, opened a new log
I20260620 23:24:48.473461 15269 ts_tablet_manager.cc:1397] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.009s
I20260620 23:24:48.473960 15266 ts_tablet_manager.cc:1428] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa: Time spent starting tablet: real 0.019s user 0.008s sys 0.008s
I20260620 23:24:48.476106 15267 tablet_bootstrap.cc:492] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c: No bootstrap required, opened a new log
I20260620 23:24:48.476405 15267 ts_tablet_manager.cc:1397] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.012s
I20260620 23:24:48.482980 15269 raft_consensus.cc:348] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:48.483371 15269 raft_consensus.cc:374] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:48.483484 15269 raft_consensus.cc:729] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b925c788004b839a529b99cdca8a3b, State: Initialized, Role: FOLLOWER
I20260620 23:24:48.483639 15267 raft_consensus.cc:348] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:48.483980 15267 raft_consensus.cc:374] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:24:48.484084 15267 raft_consensus.cc:729] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f99dbad7e7417b8b70e8f1dbc3d92c, State: Initialized, Role: FOLLOWER
I20260620 23:24:48.484175 15269 consensus_queue.cc:260] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:48.484652 15267 consensus_queue.cc:260] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:48.485260 14987 heartbeater.cc:503] Master 127.0.97.62:46597 was elected leader, sending a full tablet report...
I20260620 23:24:48.485720 15119 heartbeater.cc:503] Master 127.0.97.62:46597 was elected leader, sending a full tablet report...
I20260620 23:24:48.485895 15269 ts_tablet_manager.cc:1428] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260620 23:24:48.486320 15267 ts_tablet_manager.cc:1428] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
W20260620 23:24:48.606120 15251 tablet.cc:2369] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:24:48.641260 15120 tablet.cc:2369] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:24:48.714816 14988 tablet.cc:2369] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:24:48.940999 15274 raft_consensus.cc:482] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:24:48.941282 15274 raft_consensus.cc:504] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:48.943043 15274 leader_election.cc:288] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ffc88719f5045b49a21c29f8dc2bffa (127.0.97.3:39031), a3b925c788004b839a529b99cdca8a3b (127.0.97.1:40321)
I20260620 23:24:48.958630 15272 raft_consensus.cc:482] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:24:48.958925 15272 raft_consensus.cc:504] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:48.960814 15272 leader_election.cc:288] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28f99dbad7e7417b8b70e8f1dbc3d92c (127.0.97.2:36637), a3b925c788004b839a529b99cdca8a3b (127.0.97.1:40321)
I20260620 23:24:48.961522 15273 raft_consensus.cc:482] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:24:48.961853 15273 raft_consensus.cc:504] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:48.962796 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.943496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40321 (local address 127.0.97.2:46391)
0620 23:24:48.956016 (+ 12520us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:48.956030 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:48.956179 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:48.956516 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:48.956521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:48.956541 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:48.956854 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.956862 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.957888 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.957893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.959840 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.959853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.961711 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.961717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.961934 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.961986 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:24:48.962087 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:24:48.962245 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12383,"mutex_wait_us":46,"thread_start_us":12204,"threads_started":1}
I20260620 23:24:48.963765 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.943553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46391 (local address 127.0.97.1:40321)
0620 23:24:48.955531 (+ 11978us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:48.955536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:48.956118 (+ 582us) server_negotiation.cc:407] Connection header received
0620 23:24:48.956334 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:48.956339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:48.956394 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:48.956505 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:48.956995 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.957749 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.960025 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.961287 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.961447 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.961577 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:48.963100 (+ 1523us) server_negotiation.cc:299] Negotiation successful
0620 23:24:48.963299 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11814,"thread_start_us":12320,"threads_started":1}
I20260620 23:24:48.964141 15273 leader_election.cc:288] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ffc88719f5045b49a21c29f8dc2bffa (127.0.97.3:39031), 28f99dbad7e7417b8b70e8f1dbc3d92c (127.0.97.2:36637)
I20260620 23:24:48.965322 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.943329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32777 (local address 127.0.97.3:39031)
0620 23:24:48.955714 (+ 12385us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:48.955719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:48.955737 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:24:48.955799 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:48.955802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:48.955856 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:48.955982 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:48.956477 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.957232 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.959250 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.960423 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.960565 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.960664 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:48.964644 (+ 3980us) server_negotiation.cc:299] Negotiation successful
0620 23:24:48.964832 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12280,"thread_start_us":12190,"threads_started":1}
I20260620 23:24:48.966265 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.943312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39031 (local address 127.0.97.2:32777)
0620 23:24:48.955405 (+ 12093us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:48.955436 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:48.955598 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:48.956003 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:48.956008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:48.956036 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:48.956319 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.956327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.957361 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.957366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.959054 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.959069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.960580 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.960586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.960754 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.960807 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 23:24:48.960897 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:24:48.961045 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11917,"mutex_wait_us":22,"thread_start_us":11800,"threads_started":1}
I20260620 23:24:48.976588 14942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7114a1b70926407da8d3295bdd7424ff" candidate_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b925c788004b839a529b99cdca8a3b" is_pre_election: true
I20260620 23:24:48.977030 14942 raft_consensus.cc:2454] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28f99dbad7e7417b8b70e8f1dbc3d92c in term 0.
I20260620 23:24:48.977805 15010 leader_election.cc:302] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28f99dbad7e7417b8b70e8f1dbc3d92c, a3b925c788004b839a529b99cdca8a3b; no voters:
I20260620 23:24:48.978307 15274 raft_consensus.cc:2781] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:24:48.978439 15274 raft_consensus.cc:482] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:24:48.978535 15274 raft_consensus.cc:3037] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:48.981769 15274 raft_consensus.cc:504] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:48.982497 15274 leader_election.cc:288] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [CANDIDATE]: Term 1 election: Requested vote from peers 0ffc88719f5045b49a21c29f8dc2bffa (127.0.97.3:39031), a3b925c788004b839a529b99cdca8a3b (127.0.97.1:40321)
I20260620 23:24:48.982963 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.961579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35847 (local address 127.0.97.1:40321)
0620 23:24:48.964307 (+ 2728us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:48.964315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:48.970816 (+ 6501us) server_negotiation.cc:407] Connection header received
0620 23:24:48.972085 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:48.972093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:48.972188 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:48.972265 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:48.973415 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.974801 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.980155 (+ 5354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.981258 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.981345 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.981444 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:48.982552 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0620 23:24:48.982705 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2595,"thread_start_us":2917,"threads_started":1}
I20260620 23:24:48.983631 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.961147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40321 (local address 127.0.97.3:35847)
0620 23:24:48.970703 (+ 9556us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:48.970725 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:48.971679 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:48.972755 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:48.972763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:48.972801 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:48.973208 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.973220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.975032 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.975039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.979972 (+ 4933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.979987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.981580 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.981586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.981808 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.981840 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:24:48.983149 (+ 1309us) client_negotiation.cc:240] Negotiation successful
0620 23:24:48.983286 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9392,"mutex_wait_us":2845,"thread_start_us":13046,"threads_started":1}
I20260620 23:24:48.984097 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7114a1b70926407da8d3295bdd7424ff" candidate_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" is_pre_election: true
I20260620 23:24:48.984685 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.961628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36637 (local address 127.0.97.3:44305)
0620 23:24:48.974926 (+ 13298us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:48.974951 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:48.975047 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:48.976770 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:48.976777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:48.976817 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:48.977113 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.977126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.978998 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.979006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.981654 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.981677 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.983826 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.983831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.984021 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.984055 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:24:48.984153 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 23:24:48.984311 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13209,"thread_start_us":8806,"threads_started":1}
I20260620 23:24:48.984584 15205 raft_consensus.cc:2454] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28f99dbad7e7417b8b70e8f1dbc3d92c in term 0.
I20260620 23:24:48.985327 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.961344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44305 (local address 127.0.97.2:36637)
0620 23:24:48.976182 (+ 14838us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:48.976187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:48.976208 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:24:48.976270 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:48.976275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:48.976353 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:48.976437 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:48.977330 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.978797 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.981878 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.983419 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.983511 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.983653 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:48.984858 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0620 23:24:48.984990 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14715,"thread_start_us":14851,"threads_started":1}
I20260620 23:24:48.985456 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7114a1b70926407da8d3295bdd7424ff" candidate_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ffc88719f5045b49a21c29f8dc2bffa"
I20260620 23:24:48.985639 15205 raft_consensus.cc:3037] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:48.986210 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.974407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58463 (local address 127.0.97.3:39031)
0620 23:24:48.974543 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:48.974557 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:48.974581 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:24:48.974639 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:48.974646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:48.974787 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:48.974874 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:48.975834 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.976902 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.983919 (+ 7017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.985129 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.985226 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.985311 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:48.985882 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 23:24:48.985992 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
I20260620 23:24:48.986783 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.964780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39031 (local address 127.0.97.1:58463)
0620 23:24:48.972274 (+ 7494us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:48.972297 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:48.972438 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:48.975032 (+ 2594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:48.975036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:48.975064 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:48.975550 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.975566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.977080 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.977084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.982332 (+ 5248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.982347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.985475 (+ 3128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.985481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.985638 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.985665 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 23:24:48.986372 (+ 707us) client_negotiation.cc:240] Negotiation successful
0620 23:24:48.986507 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7329,"thread_start_us":7602,"threads_started":1}
I20260620 23:24:48.987799 15204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7114a1b70926407da8d3295bdd7424ff" candidate_uuid: "a3b925c788004b839a529b99cdca8a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" is_pre_election: true
I20260620 23:24:48.987888 14942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7114a1b70926407da8d3295bdd7424ff" candidate_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b925c788004b839a529b99cdca8a3b" is_pre_election: true
I20260620 23:24:48.988098 14942 raft_consensus.cc:2454] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ffc88719f5045b49a21c29f8dc2bffa in term 0.
I20260620 23:24:48.989250 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7114a1b70926407da8d3295bdd7424ff" candidate_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" is_pre_election: true
I20260620 23:24:48.989655 15074 raft_consensus.cc:2379] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ffc88719f5045b49a21c29f8dc2bffa in current term 1: Already voted for candidate 28f99dbad7e7417b8b70e8f1dbc3d92c in this term.
I20260620 23:24:48.989806 15141 leader_election.cc:302] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ffc88719f5045b49a21c29f8dc2bffa, a3b925c788004b839a529b99cdca8a3b; no voters:
I20260620 23:24:48.990002 15205 raft_consensus.cc:2454] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28f99dbad7e7417b8b70e8f1dbc3d92c in term 1.
I20260620 23:24:48.990585 14942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7114a1b70926407da8d3295bdd7424ff" candidate_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b925c788004b839a529b99cdca8a3b"
I20260620 23:24:48.990783 14942 raft_consensus.cc:3037] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:24:48.990900 15272 raft_consensus.cc:2741] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 23:24:48.990336 15007 leader_election.cc:302] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ffc88719f5045b49a21c29f8dc2bffa, 28f99dbad7e7417b8b70e8f1dbc3d92c; no voters:
I20260620 23:24:48.991807 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.967500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36637 (local address 127.0.97.1:60889)
0620 23:24:48.975968 (+ 8468us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:48.975987 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:24:48.976119 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:24:48.985746 (+ 9627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:24:48.985750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:24:48.985777 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:24:48.986051 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.986061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.987598 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.987602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.989249 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:24:48.989264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.991035 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.991044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:24:48.991256 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.991294 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:24:48.991389 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 23:24:48.991514 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8328,"thread_start_us":8193,"threads_started":1}
I20260620 23:24:48.992089 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:48.976385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60889 (local address 127.0.97.2:36637)
0620 23:24:48.982884 (+ 6499us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:48.982889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:48.982916 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:24:48.982967 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:48.982971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:48.983037 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:48.985505 (+ 2468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:24:48.986248 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.987466 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.989458 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:48.990665 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:48.990843 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:48.990937 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:48.991704 (+ 767us) server_negotiation.cc:299] Negotiation successful
0620 23:24:48.991832 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6372,"thread_start_us":6757,"threads_started":1}
I20260620 23:24:48.992678 15274 raft_consensus.cc:2781] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:24:48.992743 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7114a1b70926407da8d3295bdd7424ff" candidate_uuid: "a3b925c788004b839a529b99cdca8a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" is_pre_election: true
I20260620 23:24:48.993788 14942 raft_consensus.cc:2454] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28f99dbad7e7417b8b70e8f1dbc3d92c in term 1.
I20260620 23:24:48.994189 15274 raft_consensus.cc:686] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 1 LEADER]: Becoming Leader. State: Replica: 28f99dbad7e7417b8b70e8f1dbc3d92c, State: Running, Role: LEADER
I20260620 23:24:48.994750 15274 consensus_queue.cc:237] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } }
I20260620 23:24:48.998835 14876 leader_election.cc:302] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3b925c788004b839a529b99cdca8a3b; no voters: 0ffc88719f5045b49a21c29f8dc2bffa, 28f99dbad7e7417b8b70e8f1dbc3d92c
I20260620 23:24:48.999199 15273 raft_consensus.cc:2726] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 23:24:48.999795 14802 catalog_manager.cc:5495] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c reported cstate change: term changed from 0 to 1, leader changed from <none> to 28f99dbad7e7417b8b70e8f1dbc3d92c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } health_report { overall_health: UNKNOWN } } }
I20260620 23:24:49.137952 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:49.132871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.0.97.2:36637)
0620 23:24:49.133086 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:49.133090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:49.133379 (+ 289us) server_negotiation.cc:407] Connection header received
0620 23:24:49.133552 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:49.133555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:49.133656 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:49.133775 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:24:49.134139 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:49.134892 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:49.136167 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:49.136527 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:49.136614 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:49.137028 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:24:49.137515 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:24:49.137617 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:49.137700 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 23:24:49.137761 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:24:49.146006 15205 raft_consensus.cc:1261] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [term 1 FOLLOWER]: Refusing update from remote peer 28f99dbad7e7417b8b70e8f1dbc3d92c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:24:49.146044 14942 raft_consensus.cc:1261] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 1 FOLLOWER]: Refusing update from remote peer 28f99dbad7e7417b8b70e8f1dbc3d92c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:24:49.146915 15286 consensus_queue.cc:1035] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3b925c788004b839a529b99cdca8a3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:24:49.147282 15274 consensus_queue.cc:1035] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ffc88719f5045b49a21c29f8dc2bffa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:24:49.170902 14802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:24:49.171186 14802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:24:49.174985 14802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Sending DeleteTablet for 3 replicas of tablet 7114a1b70926407da8d3295bdd7424ff
I20260620 23:24:49.176050 15185 tablet_service.cc:1514] Processing DeleteTablet for tablet 7114a1b70926407da8d3295bdd7424ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:24:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34566
I20260620 23:24:49.176122 15054 tablet_service.cc:1514] Processing DeleteTablet for tablet 7114a1b70926407da8d3295bdd7424ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:24:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37690
I20260620 23:24:49.176959 15304 tablet_replica.cc:330] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa: stopping tablet replica
I20260620 23:24:49.177403 15304 raft_consensus.cc:2229] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:24:49.177436 14922 tablet_service.cc:1514] Processing DeleteTablet for tablet 7114a1b70926407da8d3295bdd7424ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:24:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43420
I20260620 23:24:49.177999 15304 raft_consensus.cc:2258] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:24:49.178763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 14839
I20260620 23:24:49.179893 15304 ts_tablet_manager.cc:1905] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:24:49.188099 15304 ts_tablet_manager.cc:1918] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:24:49.188688 15303 tablet_replica.cc:330] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c: stopping tablet replica
I20260620 23:24:49.189170 15303 raft_consensus.cc:2229] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:24:49.189450 15305 tablet_replica.cc:330] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b: stopping tablet replica
I20260620 23:24:49.189976 15303 raft_consensus.cc:2258] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:24:49.189862 15305 raft_consensus.cc:2229] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:24:49.190485 15305 raft_consensus.cc:2258] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:24:49.191902 15303 ts_tablet_manager.cc:1905] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:24:49.192274 15305 ts_tablet_manager.cc:1905] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:24:49.199067 15304 log.cc:1196] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/7114a1b70926407da8d3295bdd7424ff
I20260620 23:24:49.199270 15305 ts_tablet_manager.cc:1918] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:24:49.199429 15305 log.cc:1196] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/7114a1b70926407da8d3295bdd7424ff
I20260620 23:24:49.199647 15304 ts_tablet_manager.cc:1939] T 7114a1b70926407da8d3295bdd7424ff P 0ffc88719f5045b49a21c29f8dc2bffa: Deleting consensus metadata
I20260620 23:24:49.200001 15305 ts_tablet_manager.cc:1939] T 7114a1b70926407da8d3295bdd7424ff P a3b925c788004b839a529b99cdca8a3b: Deleting consensus metadata
I20260620 23:24:49.201715 14786 catalog_manager.cc:4841] TS 0ffc88719f5045b49a21c29f8dc2bffa (127.0.97.3:39031): tablet 7114a1b70926407da8d3295bdd7424ff (table test-table [id=b3a6bde75ab0477ea9644d8f3213961c]) successfully deleted
I20260620 23:24:49.201972 14789 catalog_manager.cc:4841] TS a3b925c788004b839a529b99cdca8a3b (127.0.97.1:40321): tablet 7114a1b70926407da8d3295bdd7424ff (table test-table [id=b3a6bde75ab0477ea9644d8f3213961c]) successfully deleted
I20260620 23:24:49.202288 15303 ts_tablet_manager.cc:1918] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:24:49.202436 15303 log.cc:1196] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/7114a1b70926407da8d3295bdd7424ff
I20260620 23:24:49.203034 15303 ts_tablet_manager.cc:1939] T 7114a1b70926407da8d3295bdd7424ff P 28f99dbad7e7417b8b70e8f1dbc3d92c: Deleting consensus metadata
I20260620 23:24:49.204623 14787 catalog_manager.cc:4841] TS 28f99dbad7e7417b8b70e8f1dbc3d92c (127.0.97.2:36637): tablet 7114a1b70926407da8d3295bdd7424ff (table test-table [id=b3a6bde75ab0477ea9644d8f3213961c]) successfully deleted
I20260620 23:24:49.206317 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:49.179905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.0.97.1:40321)
0620 23:24:49.180072 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:49.180080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:49.187334 (+ 7254us) server_negotiation.cc:407] Connection header received
0620 23:24:49.187580 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:49.187586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:49.187684 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:49.187792 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:49.190427 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:49.191613 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:49.193609 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:49.194258 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:49.194398 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:49.198978 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:49.198991 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:49.199011 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:49.199050 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:49.203235 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:49.203969 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:49.203977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:49.203986 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:49.204108 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:49.204567 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:49.204572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:49.204576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:49.205015 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:49.205298 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:49.205682 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 23:24:49.205905 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:24:49.207027 14982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:49.296696 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 14839
I20260620 23:24:49.319700 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 14991
I20260620 23:24:49.335655 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:49.320593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.0.97.2:36637)
0620 23:24:49.320777 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:49.320782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:49.320799 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:24:49.320991 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:49.320995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:49.321056 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:49.321149 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:49.323061 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:49.323830 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:49.325908 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:49.326362 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:49.326450 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:49.329772 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:49.329784 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:49.329799 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:49.329841 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:49.332952 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:49.333739 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:49.333745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:49.333747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:49.333881 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:49.334253 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:49.334256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:49.334258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:49.334603 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:49.334905 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:49.335179 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 23:24:49.335339 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:24:49.336339 15114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:49.459779 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 14991
I20260620 23:24:49.479707 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 15122
I20260620 23:24:49.493057 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:49.480483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.0.97.3:39031)
0620 23:24:49.480656 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:24:49.480660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:24:49.480677 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:24:49.480782 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:24:49.480786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:24:49.480850 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:24:49.480938 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:24:49.482138 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:49.482967 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:49.484344 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:24:49.484722 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:24:49.484814 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:24:49.487736 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:24:49.487750 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:24:49.487766 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:24:49.487797 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:24:49.490455 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:49.491238 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:49.491246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:49.491248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:49.491366 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:24:49.491832 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:24:49.491836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:24:49.491837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:24:49.492148 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:24:49.492362 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 23:24:49.492674 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 23:24:49.492833 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:24:49.493762 15245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:49.588711 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 15122
I20260620 23:24:49.604859 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 14769
I20260620 23:24:49.605988 14832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:24:49.716903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 14769
I20260620 23:24:49.738494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46597
--webserver_interface=127.0.97.62
--webserver_port=42747
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:24:50.050428 15311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:50.050774 15311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:50.050889 15311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:50.058082 15311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:24:50.058209 15311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:50.058254 15311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:50.058315 15311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:24:50.058367 15311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:24:50.067934 15311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:50.069325 15311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:50.070638 15311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:50.081961 15316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:50.081988 15317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:50.082727 15319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:50.083108 15311 server_base.cc:1034] running on GCE node
I20260620 23:24:50.083808 15311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:50.085057 15311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:50.086277 15311 hybrid_clock.cc:648] HybridClock initialized: now 1781997890086240 us; error 48 us; skew 500 ppm
I20260620 23:24:50.086596 15311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:50 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:50.089403 15311 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:24:50.090776 15311 webserver.cc:460] Webserver started at http://127.0.97.62:42747/ using document root <none> and password file <none>
I20260620 23:24:50.091413 15311 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:50.091560 15311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:50.096695 15311 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 23:24:50.099349 15326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:50.100570 15311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:24:50.100744 15311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "2596abde319b46c7a2b06674a8c3f2c9"
format_stamp: "Formatted at 2026-06-20 23:24:44 on dist-test-slave-5kzz"
server_key: "368bb5da9634f96db0e56532ddbdbf65"
server_key_iv: "f65942ebe855af550b48199d526a23b4"
server_key_version: "encryptionkey@0"
I20260620 23:24:50.101698 15311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:50.117444 15311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:50.121096 15311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:50.121366 15311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:50.138823 15311 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46597
I20260620 23:24:50.138871 15377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46597 every 8 connection(s)
I20260620 23:24:50.140241 15311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:24:50.146584 15378 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:24:50.149238 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 15311
I20260620 23:24:50.150121 15378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Bootstrap starting.
I20260620 23:24:50.150261 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40321
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46597
--webserver_port=36769
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:24:50.172473 15378 log.cc:824] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:24:50.188185 15378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:24:50.188675 15378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9: Bootstrap complete.
I20260620 23:24:50.197743 15378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } }
I20260620 23:24:50.198652 15378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2596abde319b46c7a2b06674a8c3f2c9, State: Initialized, Role: FOLLOWER
I20260620 23:24:50.199302 15378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } }
I20260620 23:24:50.199519 15378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:24:50.199625 15378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:24:50.199811 15378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:24:50.203320 15378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } }
I20260620 23:24:50.203845 15378 leader_election.cc:302] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2596abde319b46c7a2b06674a8c3f2c9; no voters:
I20260620 23:24:50.204488 15378 leader_election.cc:288] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:24:50.204629 15382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:24:50.205961 15382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [term 3 LEADER]: Becoming Leader. State: Replica: 2596abde319b46c7a2b06674a8c3f2c9, State: Running, Role: LEADER
I20260620 23:24:50.206504 15382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } }
I20260620 23:24:50.207047 15378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:24:50.209074 15384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2596abde319b46c7a2b06674a8c3f2c9. Latest consensus state: current_term: 3 leader_uuid: "2596abde319b46c7a2b06674a8c3f2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } } }
I20260620 23:24:50.209470 15384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:50.209959 15383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2596abde319b46c7a2b06674a8c3f2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2596abde319b46c7a2b06674a8c3f2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46597 } } }
I20260620 23:24:50.210391 15383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2596abde319b46c7a2b06674a8c3f2c9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:24:50.211853 15387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:24:50.219882 15387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:24:50.221305 15387 catalog_manager.cc:1259] Loaded cluster ID: 3401ee2860bd4a168eeec4ab5b478eff
I20260620 23:24:50.221463 15387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 23:24:50.223443 15387 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3fe65c09ad google::LogMessage::Fail() at ??:0
@ 0x7f3fe65c388a google::LogMessage::SendToLog() at ??:0
@ 0x7f3fe65c04de google::LogMessage::Flush() at ??:0
@ 0x7f3fe65c1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3ffbff3512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3ffbff0976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3ffc078610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3ffc078439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3fe920c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3fe920f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3fe920f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
I20260620 23:24:50.264634 15401 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f3fe91eaeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3fec4c46db start_thread at ??:0
@ 0x7f3fe545a61f clone at ??:0
W20260620 23:24:50.506987 15380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:50.507287 15380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:50.507453 15380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:50.514667 15380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:50.514869 15380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:50.515061 15380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:24:50.525312 15380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36769
--tserver_master_addrs=127.0.97.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:50.526635 15380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:50.527962 15380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:50.540426 15419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:50.540459 15422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:50.540422 15420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:50.541280 15380 server_base.cc:1034] running on GCE node
I20260620 23:24:50.541893 15380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:50.543833 15380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:50.545042 15380 hybrid_clock.cc:648] HybridClock initialized: now 1781997890544985 us; error 47 us; skew 500 ppm
I20260620 23:24:50.545356 15380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:50 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:50.548394 15380 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:24:50.549799 15380 webserver.cc:460] Webserver started at http://127.0.97.1:36769/ using document root <none> and password file <none>
I20260620 23:24:50.550437 15380 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:50.550537 15380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:50.555838 15380 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:24:50.558394 15429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:50.559702 15380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:24:50.559885 15380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "a3b925c788004b839a529b99cdca8a3b"
format_stamp: "Formatted at 2026-06-20 23:24:44 on dist-test-slave-5kzz"
server_key: "ae8c8aee91e618a594e0cd31b5f9be94"
server_key_iv: "231f70efbe16e01bc54f5c30f201f54d"
server_key_version: "encryptionkey@0"
I20260620 23:24:50.560838 15380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:50.576323 15380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:50.580159 15380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:50.580430 15380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:50.581410 15380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:50.582792 15380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:50.582901 15380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:50.583025 15380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:50.583130 15380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:50.617014 15380 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40321
I20260620 23:24:50.617074 15541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40321 every 8 connection(s)
I20260620 23:24:50.618577 15380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:24:50.623343 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:50.621463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:58533)
0620 23:24:50.622196 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:50.622391 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":303,"threads_started":1}
W20260620 23:24:50.623967 15542 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46597 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46597: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
I20260620 23:24:50.624708 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:50.624415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:56995)
0620 23:24:50.624541 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:50.624609 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:24:50.625489 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:50.625200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:50319)
0620 23:24:50.625324 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:50.625390 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 23:24:50.625713 15542 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:24:50.625789 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 15380
I20260620 23:24:50.626363 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36637
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46597
--webserver_port=45635
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:24:50.971685 15545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:50.971942 15545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:50.972090 15545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:50.979396 15545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:50.979527 15545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:50.979718 15545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:24:50.989066 15545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45635
--tserver_master_addrs=127.0.97.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:50.990357 15545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:50.991716 15545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:51.004587 15553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:51.004736 15550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:51.004717 15551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:51.005695 15545 server_base.cc:1034] running on GCE node
I20260620 23:24:51.006194 15545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:51.007135 15545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:51.008328 15545 hybrid_clock.cc:648] HybridClock initialized: now 1781997891008264 us; error 53 us; skew 500 ppm
I20260620 23:24:51.008594 15545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:51 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:51.011459 15545 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:24:51.012861 15545 webserver.cc:460] Webserver started at http://127.0.97.2:45635/ using document root <none> and password file <none>
I20260620 23:24:51.013545 15545 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:51.013676 15545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:51.019101 15545 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:24:51.021620 15560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:51.023005 15545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:24:51.023175 15545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "28f99dbad7e7417b8b70e8f1dbc3d92c"
format_stamp: "Formatted at 2026-06-20 23:24:45 on dist-test-slave-5kzz"
server_key: "8122c07c10b117dd30ac7a7a88d7e383"
server_key_iv: "5c73e3b6a97fe02f1b401afc52ad9cd8"
server_key_version: "encryptionkey@0"
I20260620 23:24:51.024147 15545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:51.038794 15545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:51.042192 15545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:51.042445 15545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:51.043473 15545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:51.044728 15545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:51.044864 15545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:51.044997 15545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:51.045092 15545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:51.077380 15545 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36637
I20260620 23:24:51.077473 15672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36637 every 8 connection(s)
I20260620 23:24:51.078898 15545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:24:51.079414 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 15545
I20260620 23:24:51.080165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39031
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46597
--webserver_port=46365
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:24:51.083969 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:51.082134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:58219)
0620 23:24:51.082949 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:51.083149 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":284,"threads_started":1}
W20260620 23:24:51.084625 15673 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46597 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46597: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
I20260620 23:24:51.085448 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:51.085163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:53083)
0620 23:24:51.085295 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:51.085354 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 23:24:51.086406 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:51.085992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:52343)
0620 23:24:51.086147 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:51.086242 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 23:24:51.086727 15673 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:24:51.392472 15676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:24:51.392755 15676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:24:51.392912 15676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:24:51.399971 15676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:24:51.400104 15676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:24:51.400293 15676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:24:51.409626 15676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46365
--tserver_master_addrs=127.0.97.62:46597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:24:51.410883 15676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:24:51.412173 15676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:24:51.424806 15681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:51.425011 15682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:24:51.424863 15684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:24:51.425737 15676 server_base.cc:1034] running on GCE node
I20260620 23:24:51.426311 15676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:24:51.427309 15676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:24:51.428519 15676 hybrid_clock.cc:648] HybridClock initialized: now 1781997891428487 us; error 43 us; skew 500 ppm
I20260620 23:24:51.428865 15676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:24:51 dist-test-slave-5kzz krb5kdc[14218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:24:51.431862 15676 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:24:51.433238 15676 webserver.cc:460] Webserver started at http://127.0.97.3:46365/ using document root <none> and password file <none>
I20260620 23:24:51.433931 15676 fs_manager.cc:359] Metadata directory not provided
I20260620 23:24:51.434072 15676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:24:51.439397 15676 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:24:51.441996 15691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:51.443316 15676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:24:51.443491 15676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "0ffc88719f5045b49a21c29f8dc2bffa"
format_stamp: "Formatted at 2026-06-20 23:24:45 on dist-test-slave-5kzz"
server_key: "5d2017dc36bb94dd12c2b1745cbd1819"
server_key_iv: "061a04182728ca123ad8ee85de838128"
server_key_version: "encryptionkey@0"
I20260620 23:24:51.444444 15676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:24:51.466323 15676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:24:51.470388 15676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:24:51.470709 15676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:24:51.471688 15676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:24:51.473156 15676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:24:51.473276 15676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:51.473415 15676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:24:51.473515 15676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:24:51.506084 15676 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39031
I20260620 23:24:51.506167 15803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39031 every 8 connection(s)
I20260620 23:24:51.507588 15676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:24:51.512185 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:51.510427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:53533)
0620 23:24:51.511153 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:51.511360 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":298,"threads_started":1}
W20260620 23:24:51.512773 15804 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46597 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46597: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
I20260620 23:24:51.513296 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 15676
I20260620 23:24:51.513674 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:51.513349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:42821)
0620 23:24:51.513508 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:51.513572 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 23:24:51.514727 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:51.514283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:37339)
0620 23:24:51.514460 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:51.514551 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:24:51.514953 15804 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:24:51.627148 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:51.626312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:37123)
0620 23:24:51.626791 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:51.626895 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":258,"threads_started":1}
I20260620 23:24:52.087826 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:52.087123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:46495)
0620 23:24:52.087534 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:52.087615 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":196,"threads_started":1}
I20260620 23:24:52.516386 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:52.515411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:60881)
0620 23:24:52.515962 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:52.516068 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":280,"threads_started":1}
I20260620 23:24:52.628633 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:52.627929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:57459)
0620 23:24:52.628362 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:52.628434 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":201,"threads_started":1}
I20260620 23:24:53.089286 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:53.088620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:42285)
0620 23:24:53.089051 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:53.089133 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":229,"threads_started":1}
I20260620 23:24:53.517892 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:53.517265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:33557)
0620 23:24:53.517679 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:53.517747 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":211,"threads_started":1}
I20260620 23:24:53.630138 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:53.629400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:55505)
0620 23:24:53.629890 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:53.629979 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":286,"threads_started":1}
I20260620 23:24:54.090934 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:54.090187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:52341)
0620 23:24:54.090620 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:54.090737 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":210,"threads_started":1}
I20260620 23:24:54.519337 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:54.518655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:52447)
0620 23:24:54.519111 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:54.519179 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":208,"threads_started":1}
I20260620 23:24:54.631572 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:54.630920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:35361)
0620 23:24:54.631342 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:54.631414 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":224,"threads_started":1}
I20260620 23:24:55.092432 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:55.091704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:37391)
0620 23:24:55.092182 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:55.092257 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":209,"threads_started":1}
I20260620 23:24:55.520823 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:55.520154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:40237)
0620 23:24:55.520567 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:55.520639 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":207,"threads_started":1}
I20260620 23:24:55.633006 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:55.632341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:60837)
0620 23:24:55.632756 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:55.632846 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":227,"threads_started":1}
I20260620 23:24:56.094051 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:56.093325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:34619)
0620 23:24:56.093798 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:56.093890 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":240,"threads_started":1}
I20260620 23:24:56.522588 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:56.521724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:37401)
0620 23:24:56.522354 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:56.522431 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":345,"threads_started":1}
I20260620 23:24:56.634560 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:56.633888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:59919)
0620 23:24:56.634327 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:56.634409 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":216,"threads_started":1}
I20260620 23:24:57.095616 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:57.094934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:40579)
0620 23:24:57.095365 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:57.095447 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":233,"threads_started":1}
I20260620 23:24:57.524152 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:57.523410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:49539)
0620 23:24:57.523872 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:57.523982 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":249,"threads_started":1}
I20260620 23:24:57.636236 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:57.635443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:43869)
0620 23:24:57.635933 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:57.636016 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":196,"threads_started":1}
I20260620 23:24:58.097358 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:58.096533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:49255)
0620 23:24:58.097048 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:58.097137 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":241,"threads_started":1}
I20260620 23:24:58.525839 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:58.525071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:33147)
0620 23:24:58.525542 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:58.525620 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":226,"threads_started":1}
I20260620 23:24:58.637907 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:58.637124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:49459)
0620 23:24:58.637636 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:58.637724 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":269,"threads_started":1}
I20260620 23:24:59.099061 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:59.098262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:47439)
0620 23:24:59.098784 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:59.098875 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":266,"threads_started":1}
I20260620 23:24:59.527467 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:59.526824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:55295)
0620 23:24:59.527236 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:59.527312 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":206,"threads_started":1}
I20260620 23:24:59.639633 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:24:59.638952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:59917)
0620 23:24:59.639389 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 23:24:59.639463 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":216,"threads_started":1}
I20260620 23:25:00.100564 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:00.099857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:54501)
0620 23:25:00.100321 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:00.100399 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
I20260620 23:25:00.529237 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:00.528356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:35715)
0620 23:25:00.528954 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:00.529064 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":319,"threads_started":1}
I20260620 23:25:00.641237 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:00.640556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:43781)
0620 23:25:00.640991 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:00.641075 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260620 23:25:01.102226 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:01.101445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:55117)
0620 23:25:01.101965 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:01.102063 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":312,"threads_started":1}
I20260620 23:25:01.530797 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:01.530108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:52241)
0620 23:25:01.530498 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:01.530570 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":214,"threads_started":1}
I20260620 23:25:01.642974 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:01.642224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:38665)
0620 23:25:01.642635 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:01.642794 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":196,"threads_started":1}
I20260620 23:25:02.104031 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:02.103227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:54713)
0620 23:25:02.103707 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:02.103828 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":234,"threads_started":1}
I20260620 23:25:02.532416 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:02.531667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:41761)
0620 23:25:02.532141 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:02.532240 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":253,"threads_started":1}
I20260620 23:25:02.644600 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:02.643899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:44791)
0620 23:25:02.644357 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:02.644439 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":243,"threads_started":1}
I20260620 23:25:03.105597 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:03.104949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:36863)
0620 23:25:03.105367 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:03.105443 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":219,"threads_started":1}
I20260620 23:25:03.533958 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:03.533299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:34305)
0620 23:25:03.533722 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:03.533805 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":208,"threads_started":1}
I20260620 23:25:03.646174 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:03.645490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:49129)
0620 23:25:03.645906 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:03.645988 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":207,"threads_started":1}
I20260620 23:25:04.107402 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:04.106474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:36175)
0620 23:25:04.107110 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:04.107194 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":391,"threads_started":1}
I20260620 23:25:04.535615 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:04.534845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:36681)
0620 23:25:04.535314 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:04.535396 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":212,"threads_started":1}
I20260620 23:25:04.647819 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:04.647062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:42763)
0620 23:25:04.647499 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:04.647608 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":201,"threads_started":1}
I20260620 23:25:05.109176 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:05.108450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:41499)
0620 23:25:05.108929 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:05.109004 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":236,"threads_started":1}
I20260620 23:25:05.537050 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:05.536404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:35205)
0620 23:25:05.536812 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:05.536883 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":214,"threads_started":1}
I20260620 23:25:05.649451 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:05.648775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.1:60661)
0620 23:25:05.649206 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:05.649288 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":233,"threads_started":1}
I20260620 23:25:06.111169 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:06.110343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.2:56395)
0620 23:25:06.110914 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:06.110991 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":342,"threads_started":1}
I20260620 23:25:06.523996 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 15380
I20260620 23:25:06.539227 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:06.538320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46597 (local address 127.0.97.3:59675)
0620 23:25:06.538881 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:06.538977 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":267,"threads_started":1}
I20260620 23:25:06.543094 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:06.525094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.0.97.1:40321)
0620 23:25:06.525821 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:06.525840 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:06.525882 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:25:06.528427 (+ 2545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:06.528440 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:06.528905 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:06.529275 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:06.530770 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:06.532007 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:06.533250 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:06.533914 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:06.534112 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:06.536820 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:06.536828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:06.536844 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:06.536884 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:06.540170 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:06.540841 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:06.540847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:06.540854 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:06.541001 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:06.541324 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:06.541327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:06.541329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:06.541825 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:06.542036 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:06.542375 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 23:25:06.542826 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":314,"threads_started":1}
I20260620 23:25:06.544425 15537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:25:06.629268 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 15380
I20260620 23:25:06.649950 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 15545
I20260620 23:25:06.668380 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:06.651084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.0.97.2:36637)
0620 23:25:06.651625 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:06.651637 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:06.651671 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:25:06.653382 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:06.653395 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:06.653888 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:06.654345 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:06.655855 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:06.657118 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:06.658354 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:06.659125 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:06.659383 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:06.661931 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:06.661939 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:06.661954 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:06.661998 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:06.665126 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:06.665849 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:06.665856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:06.665863 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:06.666024 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:06.666390 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:06.666395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:06.666398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:06.666963 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:06.667281 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:06.667679 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 23:25:06.668085 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":279,"threads_started":1}
I20260620 23:25:06.669808 15668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:25:06.750391 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 15545
I20260620 23:25:06.767652 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 15676
I20260620 23:25:06.783926 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:06.768596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47206 (local address 127.0.97.3:39031)
0620 23:25:06.769169 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:06.769183 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:06.769223 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:25:06.770849 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:06.770858 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:06.771289 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:06.771569 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:06.772738 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:06.773877 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:06.775115 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:06.775760 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:06.775891 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:06.778267 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:06.778274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:06.778287 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:06.778319 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:06.780985 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:06.781621 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:06.781626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:06.781632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:06.781783 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:06.782092 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:06.782095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:06.782097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:06.782599 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:06.782886 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:06.783239 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 23:25:06.783645 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":271,"threads_started":1}
I20260620 23:25:06.785281 15799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:25:06.865059 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 15676
2026-06-20T23:25:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24410 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:25:06 dist-test-slave-5kzz krb5kdc[15861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:25:06 dist-test-slave-5kzz krb5kdc[15861](info): set up 2 sockets
Jun 20 23:25:06 dist-test-slave-5kzz krb5kdc[15861](info): commencing operation
krb5kdc: starting...
W20260620 23:25:07.935009 388 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:25:08 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:25:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:25:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:25:08.161257 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42629
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:25:08.477885 15874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:25:08.478154 15874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:25:08.478240 15874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:25:08.485229 15874 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:25:08.485322 15874 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:25:08.485360 15874 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:25:08.485411 15874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:25:08.485464 15874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:25:08.485530 15874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:25:08.485587 15874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:25:08.485657 15874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:25:08.495416 15874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:25:08.496618 15874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:25:08.497859 15874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:25:08.509200 15879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:08.509161 15882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:08.509236 15880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:25:08.510255 15874 server_base.cc:1034] running on GCE node
I20260620 23:25:08.511097 15874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:25:08.512466 15874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:25:08.513705 15874 hybrid_clock.cc:648] HybridClock initialized: now 1781997908513640 us; error 44 us; skew 500 ppm
I20260620 23:25:08.514067 15874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:25:08 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:25:08.521756 15874 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:25:08.523279 15874 webserver.cc:460] Webserver started at http://127.0.97.62:32987/ using document root <none> and password file <none>
I20260620 23:25:08.523973 15874 fs_manager.cc:359] Metadata directory not provided
I20260620 23:25:08.524060 15874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:25:08.524330 15874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:25:08.527396 15874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "615ec1f9d0ff4f128604310003c2590e"
format_stamp: "Formatted at 2026-06-20 23:25:08 on dist-test-slave-5kzz"
server_key: "801700cfbc188126c4327238652597aa"
server_key_iv: "161477ce8cc932effb315449e3842128"
server_key_version: "encryptionkey@0"
I20260620 23:25:08.528154 15874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "615ec1f9d0ff4f128604310003c2590e"
format_stamp: "Formatted at 2026-06-20 23:25:08 on dist-test-slave-5kzz"
server_key: "801700cfbc188126c4327238652597aa"
server_key_iv: "161477ce8cc932effb315449e3842128"
server_key_version: "encryptionkey@0"
I20260620 23:25:08.533318 15874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:25:08.537070 15889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:08.538497 15874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 23:25:08.538731 15874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "615ec1f9d0ff4f128604310003c2590e"
format_stamp: "Formatted at 2026-06-20 23:25:08 on dist-test-slave-5kzz"
server_key: "801700cfbc188126c4327238652597aa"
server_key_iv: "161477ce8cc932effb315449e3842128"
server_key_version: "encryptionkey@0"
I20260620 23:25:08.538965 15874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:25:08.561323 15874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:25:08.565109 15874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:25:08.565395 15874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:25:08.588876 15874 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42629
I20260620 23:25:08.588901 15940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42629 every 8 connection(s)
I20260620 23:25:08.590775 15874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:25:08.592787 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 15874
I20260620 23:25:08.593000 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:25:08.593856 388 external_mini_cluster.cc:1428] Setting key aa3d2ae59632ab0cee1858124f0fbd80
I20260620 23:25:08.597642 15941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:25:08 dist-test-slave-5kzz krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:25:08.611840 15941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Bootstrap starting.
I20260620 23:25:08.615091 15941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Neither blocks nor log segments found. Creating new log.
I20260620 23:25:08.616178 15941 log.cc:824] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:25:08.618857 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:08.596337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.0.97.62:42629)
0620 23:25:08.597240 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:08.597256 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:08.597305 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:25:08.599864 (+ 2559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:08.599893 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:08.600286 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:08.600655 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:08.602398 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:08.603812 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:08.605863 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:08.606540 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:08.606747 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:08.611485 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:08.611494 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:08.611509 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:08.611547 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:08.614588 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:08.615410 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:08.615418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:08.615424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:08.615604 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:08.615982 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:08.615987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:08.615991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:08.616604 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:08.616988 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:08.617393 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 23:25:08.617791 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":378,"threads_started":1}
I20260620 23:25:08.619360 15941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: No bootstrap required, opened a new log
I20260620 23:25:08.628213 15941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:25:08.628794 15941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:25:08.628911 15941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615ec1f9d0ff4f128604310003c2590e, State: Initialized, Role: FOLLOWER
I20260620 23:25:08.629611 15941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:25:08.629889 15941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:25:08.630100 15941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:25:08.630306 15941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:25:08.633122 15941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:25:08.633664 15941 leader_election.cc:302] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 615ec1f9d0ff4f128604310003c2590e; no voters:
I20260620 23:25:08.634241 15941 leader_election.cc:288] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:25:08.634382 15946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:25:08.635102 15946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 1 LEADER]: Becoming Leader. State: Replica: 615ec1f9d0ff4f128604310003c2590e, State: Running, Role: LEADER
I20260620 23:25:08.635672 15946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:25:08.636433 15941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:25:08.638751 15947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "615ec1f9d0ff4f128604310003c2590e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } } }
I20260620 23:25:08.639858 15947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: This master's current role is: LEADER
I20260620 23:25:08.639981 15948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 615ec1f9d0ff4f128604310003c2590e. Latest consensus state: current_term: 1 leader_uuid: "615ec1f9d0ff4f128604310003c2590e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } } }
I20260620 23:25:08.643304 15948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: This master's current role is: LEADER
I20260620 23:25:08.648398 15953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:25:08.652654 15953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:25:08.661044 15953 catalog_manager.cc:1347] Generated new cluster ID: a76e42edc22b4728b78d13ac41ff1e21
I20260620 23:25:08.661168 15953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:25:08.673342 15953 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:25:08.674567 15953 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:25:08.683979 15953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Generated new TSK 0
I20260620 23:25:08.684788 15953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:25:08.750032 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42629
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:25:09.073621 15967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:25:09.073884 15967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:25:09.074038 15967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:25:09.081123 15967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:25:09.081223 15967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:25:09.081413 15967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:25:09.090580 15967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:25:09.091776 15967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:25:09.093044 15967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:25:09.105861 15975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:09.105937 15973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:09.105899 15972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:25:09.106959 15967 server_base.cc:1034] running on GCE node
I20260620 23:25:09.107575 15967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:25:09.108443 15967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:25:09.109664 15967 hybrid_clock.cc:648] HybridClock initialized: now 1781997909109606 us; error 42 us; skew 500 ppm
I20260620 23:25:09.109959 15967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:25:09 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:25:09.112928 15967 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:25:09.114173 15967 webserver.cc:460] Webserver started at http://127.0.97.1:35053/ using document root <none> and password file <none>
I20260620 23:25:09.114900 15967 fs_manager.cc:359] Metadata directory not provided
I20260620 23:25:09.115000 15967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:25:09.115250 15967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:25:09.118026 15967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "31bb8d884058485385ecbe841f6a0a72"
format_stamp: "Formatted at 2026-06-20 23:25:09 on dist-test-slave-5kzz"
server_key: "67279e6e0b45fc773131aea6e853f662"
server_key_iv: "0e87925a48228f9dab7d81628d01be5a"
server_key_version: "encryptionkey@0"
I20260620 23:25:09.118762 15967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "31bb8d884058485385ecbe841f6a0a72"
format_stamp: "Formatted at 2026-06-20 23:25:09 on dist-test-slave-5kzz"
server_key: "67279e6e0b45fc773131aea6e853f662"
server_key_iv: "0e87925a48228f9dab7d81628d01be5a"
server_key_version: "encryptionkey@0"
I20260620 23:25:09.123557 15967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:25:09.126636 15982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:09.127779 15967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 23:25:09.127887 15967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "31bb8d884058485385ecbe841f6a0a72"
format_stamp: "Formatted at 2026-06-20 23:25:09 on dist-test-slave-5kzz"
server_key: "67279e6e0b45fc773131aea6e853f662"
server_key_iv: "0e87925a48228f9dab7d81628d01be5a"
server_key_version: "encryptionkey@0"
I20260620 23:25:09.128026 15967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:25:09.156952 15967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:25:09.160764 15967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:25:09.160997 15967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:25:09.161824 15967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:25:09.163120 15967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:25:09.163193 15967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:09.163285 15967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:25:09.163334 15967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:09.195521 15967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45227
I20260620 23:25:09.195569 16094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45227 every 8 connection(s)
I20260620 23:25:09.196812 15967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:25:09.204622 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 15967
I20260620 23:25:09.204835 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:25:09.205467 388 external_mini_cluster.cc:1428] Setting key 4d0db444216fd65d1b1b848cc279dc48
Jun 20 23:25:09 dist-test-slave-5kzz krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:25:09.216250 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:09.199266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41705 (local address 127.0.97.62:42629)
0620 23:25:09.199657 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:09.199662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:09.200565 (+ 903us) server_negotiation.cc:407] Connection header received
0620 23:25:09.202206 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:09.202210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:09.202272 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:09.202367 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:09.203783 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:09.204819 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:09.206532 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:09.206968 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:09.207099 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:09.210542 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:09.210547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:09.210550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:09.210580 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:09.213160 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:09.213884 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:09.213890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:09.213893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:09.214015 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:09.214424 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:09.214431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:09.214436 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:09.214787 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:09.214934 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:09.215668 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 23:25:09.215899 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":179,"threads_started":1}
I20260620 23:25:09.217159 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:09.199784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:41705)
0620 23:25:09.200412 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:09.200449 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:09.201975 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:09.202552 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:09.202559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:09.203070 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:25:09.203623 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:09.203639 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:09.204996 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:09.204999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:09.206393 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:09.206404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:09.207181 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:09.207190 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:09.207425 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:09.208077 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:25:09.208098 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:25:09.210336 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:25:09.213331 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:09.213340 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:09.213357 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:09.213764 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:09.214119 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:09.214125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:09.214129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:09.214319 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:09.214917 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:25:09.214926 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:25:09.215267 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 23:25:09.215562 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 23:25:09.216092 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":257,"threads_started":1}
I20260620 23:25:09.218776 16095 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42629
I20260620 23:25:09.219063 16095 heartbeater.cc:464] Registering TS with master...
I20260620 23:25:09.219732 16095 heartbeater.cc:511] Master 127.0.97.62:42629 requested a full tablet report, sending...
I20260620 23:25:09.222002 15906 ts_manager.cc:194] Registered new tserver with Master: 31bb8d884058485385ecbe841f6a0a72 (127.0.97.1:45227)
I20260620 23:25:09.224210 15906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41705
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:25:09.257002 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42629
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:25:09.567566 16101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:25:09.567822 16101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:25:09.567988 16101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:25:09.577253 16101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:25:09.577375 16101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:25:09.577528 16101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:25:09.586826 16101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:25:09.588025 16101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:25:09.589313 16101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:25:09.601763 16106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:09.601728 16107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:09.602290 16109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:25:09.602640 16101 server_base.cc:1034] running on GCE node
I20260620 23:25:09.603255 16101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:25:09.604104 16101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:25:09.605312 16101 hybrid_clock.cc:648] HybridClock initialized: now 1781997909605263 us; error 44 us; skew 500 ppm
I20260620 23:25:09.605599 16101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:25:09 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:25:09.608654 16101 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:25:09.609975 16101 webserver.cc:460] Webserver started at http://127.0.97.2:33553/ using document root <none> and password file <none>
I20260620 23:25:09.610625 16101 fs_manager.cc:359] Metadata directory not provided
I20260620 23:25:09.610795 16101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:25:09.611044 16101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:25:09.613662 16101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "3a66b333277a422093df07050647b342"
format_stamp: "Formatted at 2026-06-20 23:25:09 on dist-test-slave-5kzz"
server_key: "53ff8f437cfedc07a9feacf523447367"
server_key_iv: "222d29b7e51dddce2530adabfc4742df"
server_key_version: "encryptionkey@0"
I20260620 23:25:09.614334 16101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3a66b333277a422093df07050647b342"
format_stamp: "Formatted at 2026-06-20 23:25:09 on dist-test-slave-5kzz"
server_key: "53ff8f437cfedc07a9feacf523447367"
server_key_iv: "222d29b7e51dddce2530adabfc4742df"
server_key_version: "encryptionkey@0"
I20260620 23:25:09.619292 16101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 23:25:09.622473 16116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:09.623687 16101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:25:09.623823 16101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "3a66b333277a422093df07050647b342"
format_stamp: "Formatted at 2026-06-20 23:25:09 on dist-test-slave-5kzz"
server_key: "53ff8f437cfedc07a9feacf523447367"
server_key_iv: "222d29b7e51dddce2530adabfc4742df"
server_key_version: "encryptionkey@0"
I20260620 23:25:09.623975 16101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:25:09.650795 16101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:25:09.654937 16101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:25:09.655161 16101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:25:09.656065 16101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:25:09.657230 16101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:25:09.657296 16101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:09.657384 16101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:25:09.657424 16101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:09.688476 16101 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40429
I20260620 23:25:09.688531 16228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40429 every 8 connection(s)
I20260620 23:25:09.689694 16101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:25:09.690215 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 16101
I20260620 23:25:09.690404 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:25:09.691040 388 external_mini_cluster.cc:1428] Setting key 79d5a56956d4f62d83d486df096e594d
Jun 20 23:25:09 dist-test-slave-5kzz krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:25:09.709300 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:09.692327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54953 (local address 127.0.97.62:42629)
0620 23:25:09.692493 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:09.692498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:09.693894 (+ 1396us) server_negotiation.cc:407] Connection header received
0620 23:25:09.695775 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:09.695783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:09.695873 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:09.695974 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:09.697404 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:09.698501 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:09.699988 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:09.700406 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:09.700529 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:09.703893 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:09.703898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:09.703901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:09.703931 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:09.706395 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:09.707128 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:09.707133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:09.707136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:09.707258 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:09.707654 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:09.707659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:09.707662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:09.707913 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:09.708060 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:09.708729 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 23:25:09.708963 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:25:09.710243 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:09.692946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:54953)
0620 23:25:09.693733 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:09.693777 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:09.695510 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:09.696182 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:09.696190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:09.696671 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:25:09.697224 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:09.697241 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:09.698624 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:09.698627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:09.699852 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:09.699867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:09.700621 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:09.700629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:09.700860 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:09.701505 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:25:09.701525 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:25:09.703691 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:25:09.706530 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:09.706538 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:09.706554 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:09.707012 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:09.707364 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:09.707370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:09.707373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:09.707544 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:09.708029 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:25:09.708037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:25:09.708365 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 23:25:09.708630 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 23:25:09.709141 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":330,"threads_started":1}
I20260620 23:25:09.711853 16229 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42629
I20260620 23:25:09.712159 16229 heartbeater.cc:464] Registering TS with master...
I20260620 23:25:09.712849 16229 heartbeater.cc:511] Master 127.0.97.62:42629 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:25:09.714242 15906 ts_manager.cc:194] Registered new tserver with Master: 3a66b333277a422093df07050647b342 (127.0.97.2:40429)
I20260620 23:25:09.715152 15906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:25:09.740298 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42629
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:25:10.057673 16234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:25:10.057964 16234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:25:10.058060 16234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:25:10.065016 16234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:25:10.065122 16234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:25:10.065235 16234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:25:10.074936 16234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:25:10.076200 16234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:25:10.077435 16234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:25:10.088989 16239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:10.089272 16240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:10.090081 16242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:25:10.090495 16234 server_base.cc:1034] running on GCE node
I20260620 23:25:10.091094 16234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:25:10.092018 16234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:25:10.093326 16234 hybrid_clock.cc:648] HybridClock initialized: now 1781997910093273 us; error 56 us; skew 500 ppm
I20260620 23:25:10.093662 16234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:25:10 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:25:10.096838 16234 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:25:10.098237 16234 webserver.cc:460] Webserver started at http://127.0.97.3:43255/ using document root <none> and password file <none>
I20260620 23:25:10.098913 16234 fs_manager.cc:359] Metadata directory not provided
I20260620 23:25:10.099047 16234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:25:10.099335 16234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:25:10.102180 16234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "935fe28ae7fb4ef4bababd665351bbb9"
format_stamp: "Formatted at 2026-06-20 23:25:10 on dist-test-slave-5kzz"
server_key: "86fe8ab194fb68f380e5e4ec4164b50f"
server_key_iv: "c6995a01ef20a211140ea043446251e3"
server_key_version: "encryptionkey@0"
I20260620 23:25:10.102979 16234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "935fe28ae7fb4ef4bababd665351bbb9"
format_stamp: "Formatted at 2026-06-20 23:25:10 on dist-test-slave-5kzz"
server_key: "86fe8ab194fb68f380e5e4ec4164b50f"
server_key_iv: "c6995a01ef20a211140ea043446251e3"
server_key_version: "encryptionkey@0"
I20260620 23:25:10.107878 16234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:25:10.111186 16249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:10.112560 16234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:25:10.112735 16234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "935fe28ae7fb4ef4bababd665351bbb9"
format_stamp: "Formatted at 2026-06-20 23:25:10 on dist-test-slave-5kzz"
server_key: "86fe8ab194fb68f380e5e4ec4164b50f"
server_key_iv: "c6995a01ef20a211140ea043446251e3"
server_key_version: "encryptionkey@0"
I20260620 23:25:10.112931 16234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:25:10.138509 16234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:25:10.142930 16234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:25:10.143204 16234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:25:10.144191 16234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:25:10.145525 16234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:25:10.145622 16234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:10.145769 16234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:25:10.145836 16234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:10.178164 16234 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46745
I20260620 23:25:10.178212 16361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46745 every 8 connection(s)
I20260620 23:25:10.179533 16234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:25:10.181146 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 16234
I20260620 23:25:10.181347 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:25:10.181806 388 external_mini_cluster.cc:1428] Setting key acd4a09bbed142d9aacfcec66b4e9f25
Jun 20 23:25:10 dist-test-slave-5kzz krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:25:10.199644 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.182022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55887 (local address 127.0.97.62:42629)
0620 23:25:10.182243 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:10.182250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:10.183645 (+ 1395us) server_negotiation.cc:407] Connection header received
0620 23:25:10.185735 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:10.185739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:10.185803 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:10.185887 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:10.187373 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.188141 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.189563 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.189933 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.190019 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.193233 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:10.193238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:10.193241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:10.193270 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:10.196591 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:10.197456 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:10.197460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:10.197462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:10.197583 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:10.198027 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:10.198030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:10.198032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:10.198283 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:10.198396 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:10.199243 (+ 847us) server_negotiation.cc:299] Negotiation successful
0620 23:25:10.199404 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:25:10.200913 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.182560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:55887)
0620 23:25:10.183470 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:10.183521 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:10.185473 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:10.186090 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:10.186098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:10.186603 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:25:10.187226 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:10.187243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.188256 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.188259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:10.189435 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:10.189446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.190103 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.190107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:10.190290 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.190937 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:25:10.190959 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:25:10.193005 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:25:10.196717 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:10.196726 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:10.196741 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:10.197325 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:10.197683 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:10.197687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:10.197691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:10.197920 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:10.198416 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:25:10.198424 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:25:10.198843 (+ 419us) client_negotiation.cc:769] Sending connection context
0620 23:25:10.199143 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 23:25:10.199638 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":314,"threads_started":1}
I20260620 23:25:10.202217 16362 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42629
I20260620 23:25:10.202525 16362 heartbeater.cc:464] Registering TS with master...
I20260620 23:25:10.203241 16362 heartbeater.cc:511] Master 127.0.97.62:42629 requested a full tablet report, sending...
I20260620 23:25:10.204559 15906 ts_manager.cc:194] Registered new tserver with Master: 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745)
I20260620 23:25:10.205374 15906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55887
I20260620 23:25:10.209194 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:25:10.225149 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.213560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.0.97.62:42629)
0620 23:25:10.213757 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:10.213766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:10.214115 (+ 349us) server_negotiation.cc:407] Connection header received
0620 23:25:10.214242 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:10.214245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:10.214311 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:10.214389 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:10.215782 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.216539 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.217893 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.218278 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.218377 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.220622 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:10.220627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:10.220630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:10.220659 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:10.222985 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:10.223560 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:10.223563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:10.223565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:10.223686 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:10.223966 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:10.223968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:10.223970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:10.224246 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:10.224444 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:10.224667 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 23:25:10.224831 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:25:10.227236 16095 heartbeater.cc:503] Master 127.0.97.62:42629 was elected leader, sending a full tablet report...
I20260620 23:25:10.229483 15906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:25:10.236343 15906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:25:10.272935 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.248379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46745 (local address 127.0.0.1:47564)
0620 23:25:10.263563 (+ 15184us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:10.263588 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:10.263750 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:10.264940 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:10.264948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:10.264979 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:25:10.265393 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:10.265409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.267529 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.267534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:10.270130 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:10.270148 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.272105 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.272111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:10.272307 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.272345 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:25:10.272441 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:25:10.272594 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15009,"mutex_wait_us":40,"thread_start_us":10641,"threads_started":1}
I20260620 23:25:10.273563 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.248328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40429 (local address 127.0.0.1:43952)
0620 23:25:10.259431 (+ 11103us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:10.259471 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:10.259664 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:10.266006 (+ 6342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:10.266009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:10.266029 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:25:10.266313 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:10.266320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.268205 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.268209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:10.270407 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:10.270417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.272504 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.272528 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:10.272824 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.272911 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 23:25:10.273025 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:25:10.273164 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10845,"mutex_wait_us":153,"thread_start_us":16125,"threads_started":1}
I20260620 23:25:10.273761 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.248822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.0.97.3:46745)
0620 23:25:10.262817 (+ 13995us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:10.262830 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:10.263681 (+ 851us) server_negotiation.cc:407] Connection header received
0620 23:25:10.263939 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:10.263952 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:10.264283 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:10.265582 (+ 1299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:25:10.265919 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.267385 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.270297 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.271435 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.271555 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.273184 (+ 1629us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:10.273260 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 23:25:10.273426 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13767,"thread_start_us":13584,"threads_started":1}
I20260620 23:25:10.273803 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.248511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.0.97.2:40429)
0620 23:25:10.264577 (+ 16066us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:10.264588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:10.264615 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:25:10.264719 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:10.264727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:10.264973 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:10.265188 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:25:10.266478 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.268029 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.270544 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.272016 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.272140 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.272331 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:10.273350 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0620 23:25:10.273501 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15899,"thread_start_us":16876,"threads_started":1}
I20260620 23:25:10.273887 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.260568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.0.97.1:45227)
0620 23:25:10.266133 (+ 5565us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:10.266142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:10.266167 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:25:10.266274 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:10.266285 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:10.266503 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:10.266701 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:25:10.267288 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.268736 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.270704 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.272016 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.272140 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.272349 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:10.273380 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0620 23:25:10.273555 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5390,"thread_start_us":5217,"threads_started":1}
I20260620 23:25:10.274446 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.259956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45227 (local address 127.0.0.1:60700)
0620 23:25:10.264497 (+ 4541us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:10.264518 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:10.264648 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:10.266717 (+ 2069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:10.266722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:10.266746 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:25:10.267102 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:10.267113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.268905 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.268909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:10.270534 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:10.270548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.272536 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.272542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:10.272865 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.272936 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 23:25:10.273011 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:25:10.274163 (+ 1152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4367,"mutex_wait_us":120,"thread_start_us":3912,"threads_started":1}
I20260620 23:25:10.277488 16164 tablet_service.cc:1467] Processing CreateTablet for tablet 83e0ff7a1e444fd0913ca1df0cb8df6f (DEFAULT_TABLE table=test-table [id=d92a700b437142848427e91b08121f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:25:10.277688 16297 tablet_service.cc:1467] Processing CreateTablet for tablet 83e0ff7a1e444fd0913ca1df0cb8df6f (DEFAULT_TABLE table=test-table [id=d92a700b437142848427e91b08121f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:25:10.277694 16030 tablet_service.cc:1467] Processing CreateTablet for tablet 83e0ff7a1e444fd0913ca1df0cb8df6f (DEFAULT_TABLE table=test-table [id=d92a700b437142848427e91b08121f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:25:10.278748 16164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e0ff7a1e444fd0913ca1df0cb8df6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:25:10.279090 16297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e0ff7a1e444fd0913ca1df0cb8df6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:25:10.279088 16030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e0ff7a1e444fd0913ca1df0cb8df6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:25:10.290930 16380 tablet_bootstrap.cc:492] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342: Bootstrap starting.
I20260620 23:25:10.293012 16381 tablet_bootstrap.cc:492] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9: Bootstrap starting.
I20260620 23:25:10.294390 16382 tablet_bootstrap.cc:492] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72: Bootstrap starting.
I20260620 23:25:10.294618 16380 tablet_bootstrap.cc:654] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342: Neither blocks nor log segments found. Creating new log.
I20260620 23:25:10.295967 16380 log.cc:824] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342: Log is configured to *not* fsync() on all Append() calls
I20260620 23:25:10.296468 16381 tablet_bootstrap.cc:654] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9: Neither blocks nor log segments found. Creating new log.
I20260620 23:25:10.297439 16382 tablet_bootstrap.cc:654] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72: Neither blocks nor log segments found. Creating new log.
I20260620 23:25:10.297886 16381 log.cc:824] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:25:10.298446 16382 log.cc:824] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72: Log is configured to *not* fsync() on all Append() calls
I20260620 23:25:10.300979 16381 tablet_bootstrap.cc:492] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9: No bootstrap required, opened a new log
I20260620 23:25:10.301250 16381 ts_tablet_manager.cc:1397] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 23:25:10.301445 16382 tablet_bootstrap.cc:492] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72: No bootstrap required, opened a new log
I20260620 23:25:10.301501 16380 tablet_bootstrap.cc:492] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342: No bootstrap required, opened a new log
I20260620 23:25:10.301678 16382 ts_tablet_manager.cc:1397] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260620 23:25:10.301741 16380 ts_tablet_manager.cc:1397] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 23:25:10.309088 16382 raft_consensus.cc:348] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } } peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } }
I20260620 23:25:10.309448 16382 raft_consensus.cc:374] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:25:10.309543 16382 raft_consensus.cc:729] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31bb8d884058485385ecbe841f6a0a72, State: Initialized, Role: FOLLOWER
I20260620 23:25:10.310151 16382 consensus_queue.cc:260] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } } peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } }
I20260620 23:25:10.311923 16382 ts_tablet_manager.cc:1428] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260620 23:25:10.311916 16381 raft_consensus.cc:348] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } } peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } }
I20260620 23:25:10.312325 16381 raft_consensus.cc:374] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:25:10.312423 16381 raft_consensus.cc:729] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935fe28ae7fb4ef4bababd665351bbb9, State: Initialized, Role: FOLLOWER
I20260620 23:25:10.312976 16381 consensus_queue.cc:260] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } } peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } }
I20260620 23:25:10.313946 16362 heartbeater.cc:503] Master 127.0.97.62:42629 was elected leader, sending a full tablet report...
I20260620 23:25:10.314630 16381 ts_tablet_manager.cc:1428] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 23:25:10.318451 16380 raft_consensus.cc:348] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } } peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } }
I20260620 23:25:10.318811 16380 raft_consensus.cc:374] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:25:10.318938 16380 raft_consensus.cc:729] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a66b333277a422093df07050647b342, State: Initialized, Role: FOLLOWER
I20260620 23:25:10.319449 16380 consensus_queue.cc:260] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } } peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } }
I20260620 23:25:10.320432 16229 heartbeater.cc:503] Master 127.0.97.62:42629 was elected leader, sending a full tablet report...
I20260620 23:25:10.320962 16380 ts_tablet_manager.cc:1428] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342: Time spent starting tablet: real 0.019s user 0.014s sys 0.000s
I20260620 23:25:10.328339 16386 raft_consensus.cc:482] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:25:10.328578 16386 raft_consensus.cc:504] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } } peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } }
I20260620 23:25:10.330367 16386 leader_election.cc:288] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a66b333277a422093df07050647b342 (127.0.97.2:40429), 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745)
I20260620 23:25:10.336942 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.330640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52273 (local address 127.0.97.2:40429)
0620 23:25:10.330858 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:10.330867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:10.331820 (+ 953us) server_negotiation.cc:407] Connection header received
0620 23:25:10.331957 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:10.331964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:10.332033 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:10.332126 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:25:10.332789 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.333474 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.335266 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.336008 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.336164 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.336251 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:10.336582 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 23:25:10.336705 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:25:10.336952 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.331067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37625 (local address 127.0.97.3:46745)
0620 23:25:10.331256 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:10.331262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:10.331572 (+ 310us) server_negotiation.cc:407] Connection header received
0620 23:25:10.331797 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:10.331803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:10.331875 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:10.331959 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:25:10.332633 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.333388 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.335208 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.335965 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.336061 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.336172 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:10.336542 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 23:25:10.336668 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:25:10.337086 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.330759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40429 (local address 127.0.97.1:52273)
0620 23:25:10.331691 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:10.331720 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:10.331838 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:10.332149 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:10.332155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:10.332181 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:25:10.332631 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:10.332643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.333606 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.333612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:10.335106 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:10.335118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.336260 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.336265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:10.336418 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.336440 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:25:10.336545 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:25:10.336669 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":41,"thread_start_us":617,"threads_started":1}
I20260620 23:25:10.337463 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.330727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46745 (local address 127.0.97.1:37625)
0620 23:25:10.331456 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:10.331482 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:10.331634 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:10.331990 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:10.331994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:10.332016 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:25:10.332486 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:10.332499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.333505 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.333513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:10.335073 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:10.335084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.336176 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.336185 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:10.336395 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.336419 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:25:10.336491 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 23:25:10.336627 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":20,"thread_start_us":310,"threads_started":1}
I20260620 23:25:10.338012 16184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e0ff7a1e444fd0913ca1df0cb8df6f" candidate_uuid: "31bb8d884058485385ecbe841f6a0a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a66b333277a422093df07050647b342" is_pre_election: true
I20260620 23:25:10.338292 16317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e0ff7a1e444fd0913ca1df0cb8df6f" candidate_uuid: "31bb8d884058485385ecbe841f6a0a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935fe28ae7fb4ef4bababd665351bbb9" is_pre_election: true
I20260620 23:25:10.338438 16184 raft_consensus.cc:2454] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31bb8d884058485385ecbe841f6a0a72 in term 0.
I20260620 23:25:10.338778 16317 raft_consensus.cc:2454] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31bb8d884058485385ecbe841f6a0a72 in term 0.
I20260620 23:25:10.339119 15985 leader_election.cc:302] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31bb8d884058485385ecbe841f6a0a72, 3a66b333277a422093df07050647b342; no voters:
I20260620 23:25:10.339483 16386 raft_consensus.cc:2781] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:25:10.339627 16386 raft_consensus.cc:482] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:25:10.339728 16386 raft_consensus.cc:3037] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:25:10.341773 16386 raft_consensus.cc:504] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } } peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } }
I20260620 23:25:10.342346 16386 leader_election.cc:288] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 1 election: Requested vote from peers 3a66b333277a422093df07050647b342 (127.0.97.2:40429), 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745)
I20260620 23:25:10.342645 16184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e0ff7a1e444fd0913ca1df0cb8df6f" candidate_uuid: "31bb8d884058485385ecbe841f6a0a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a66b333277a422093df07050647b342"
I20260620 23:25:10.342823 16317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e0ff7a1e444fd0913ca1df0cb8df6f" candidate_uuid: "31bb8d884058485385ecbe841f6a0a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935fe28ae7fb4ef4bababd665351bbb9"
I20260620 23:25:10.342887 16184 raft_consensus.cc:3037] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:25:10.343027 16317 raft_consensus.cc:3037] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:25:10.345513 16184 raft_consensus.cc:2454] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31bb8d884058485385ecbe841f6a0a72 in term 1.
I20260620 23:25:10.345782 16317 raft_consensus.cc:2454] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31bb8d884058485385ecbe841f6a0a72 in term 1.
I20260620 23:25:10.345871 15985 leader_election.cc:302] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31bb8d884058485385ecbe841f6a0a72, 3a66b333277a422093df07050647b342; no voters:
I20260620 23:25:10.346120 16386 raft_consensus.cc:2781] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:25:10.346549 16386 raft_consensus.cc:686] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 1 LEADER]: Becoming Leader. State: Replica: 31bb8d884058485385ecbe841f6a0a72, State: Running, Role: LEADER
I20260620 23:25:10.347004 16386 consensus_queue.cc:237] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } } peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } }
I20260620 23:25:10.351320 15905 catalog_manager.cc:5495] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31bb8d884058485385ecbe841f6a0a72 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "31bb8d884058485385ecbe841f6a0a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } health_report { overall_health: HEALTHY } } }
I20260620 23:25:10.375800 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.370789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.0.97.1:45227)
0620 23:25:10.370992 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:10.370996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:10.371014 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:25:10.371166 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:10.371170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:10.371259 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:10.371376 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:25:10.371699 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.372386 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.373861 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.374237 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.374329 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.374792 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:25:10.375272 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:25:10.375390 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:10.375455 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 23:25:10.375512 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:25:10.383625 16184 raft_consensus.cc:1261] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342 [term 1 FOLLOWER]: Refusing update from remote peer 31bb8d884058485385ecbe841f6a0a72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:25:10.383625 16317 raft_consensus.cc:1261] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9 [term 1 FOLLOWER]: Refusing update from remote peer 31bb8d884058485385ecbe841f6a0a72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:25:10.384668 16391 consensus_queue.cc:1035] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:25:10.385031 16386 consensus_queue.cc:1035] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:25:10.396586 16396 mvcc.cc:205] Tried to move back new op lower bound from 7299063440920711168 to 7299063440784863232. Current Snapshot: MvccSnapshot[applied={T|T < 7299063440920711168}]
I20260620 23:25:10.408378 15906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:25:10.408654 15906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:25:10.412094 15906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Sending DeleteTablet for 3 replicas of tablet 83e0ff7a1e444fd0913ca1df0cb8df6f
I20260620 23:25:10.413156 16164 tablet_service.cc:1514] Processing DeleteTablet for tablet 83e0ff7a1e444fd0913ca1df0cb8df6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:25:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43952
I20260620 23:25:10.413484 16297 tablet_service.cc:1514] Processing DeleteTablet for tablet 83e0ff7a1e444fd0913ca1df0cb8df6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:25:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47564
I20260620 23:25:10.413494 16030 tablet_service.cc:1514] Processing DeleteTablet for tablet 83e0ff7a1e444fd0913ca1df0cb8df6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:25:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60700
I20260620 23:25:10.414450 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 15967
I20260620 23:25:10.423399 16405 tablet_replica.cc:330] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9: stopping tablet replica
I20260620 23:25:10.423930 16405 raft_consensus.cc:2229] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:25:10.424515 16405 raft_consensus.cc:2258] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:25:10.429075 16405 ts_tablet_manager.cc:1905] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:25:10.436859 16404 tablet_replica.cc:330] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72: stopping tablet replica
I20260620 23:25:10.437458 16404 raft_consensus.cc:2229] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:25:10.438271 16404 raft_consensus.cc:2258] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 23:25:10 dist-test-slave-5kzz krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:25:10.442369 16403 tablet_replica.cc:330] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342: stopping tablet replica
I20260620 23:25:10.443148 16403 raft_consensus.cc:2229] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:25:10.443648 16403 raft_consensus.cc:2258] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:25:10.443928 16405 ts_tablet_manager.cc:1918] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:25:10.444097 16405 log.cc:1196] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/83e0ff7a1e444fd0913ca1df0cb8df6f
I20260620 23:25:10.444715 16405 ts_tablet_manager.cc:1939] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 935fe28ae7fb4ef4bababd665351bbb9: Deleting consensus metadata
I20260620 23:25:10.445322 16404 ts_tablet_manager.cc:1905] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:25:10.445652 16403 ts_tablet_manager.cc:1905] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:25:10.446762 15891 catalog_manager.cc:4841] TS 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745): tablet 83e0ff7a1e444fd0913ca1df0cb8df6f (table test-table [id=d92a700b437142848427e91b08121f8d]) successfully deleted
I20260620 23:25:10.452579 16403 ts_tablet_manager.cc:1918] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:25:10.452740 16403 log.cc:1196] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/83e0ff7a1e444fd0913ca1df0cb8df6f
I20260620 23:25:10.453238 16403 ts_tablet_manager.cc:1939] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 3a66b333277a422093df07050647b342: Deleting consensus metadata
I20260620 23:25:10.455080 15892 catalog_manager.cc:4841] TS 3a66b333277a422093df07050647b342 (127.0.97.2:40429): tablet 83e0ff7a1e444fd0913ca1df0cb8df6f (table test-table [id=d92a700b437142848427e91b08121f8d]) successfully deleted
I20260620 23:25:10.455132 16404 ts_tablet_manager.cc:1918] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:25:10.455292 16404 log.cc:1196] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/83e0ff7a1e444fd0913ca1df0cb8df6f
I20260620 23:25:10.455878 16404 ts_tablet_manager.cc:1939] T 83e0ff7a1e444fd0913ca1df0cb8df6f P 31bb8d884058485385ecbe841f6a0a72: Deleting consensus metadata
I20260620 23:25:10.457479 15891 catalog_manager.cc:4841] TS 31bb8d884058485385ecbe841f6a0a72 (127.0.97.1:45227): tablet 83e0ff7a1e444fd0913ca1df0cb8df6f (table test-table [id=d92a700b437142848427e91b08121f8d]) successfully deleted
I20260620 23:25:10.458052 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.415833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.0.97.1:45227)
0620 23:25:10.416051 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:10.416060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:10.428967 (+ 12907us) server_negotiation.cc:407] Connection header received
0620 23:25:10.429475 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:10.429481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:10.429559 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:10.429637 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:10.432075 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.433670 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.435842 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.436752 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.436860 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.449980 (+ 13120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:10.449994 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:10.450010 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:10.450058 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:10.454459 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:10.455701 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:10.455711 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:10.455714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:10.455915 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:10.456350 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:10.456355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:10.456358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:10.456731 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:10.457059 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:10.457488 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 23:25:10.457790 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:25:10.458734 16090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:25:10.555963 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 15967
I20260620 23:25:10.574525 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 16101
Jun 20 23:25:10 dist-test-slave-5kzz krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:25:10.588891 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.575275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.0.97.2:40429)
0620 23:25:10.575452 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:10.575456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:10.575474 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:25:10.575658 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:10.575662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:10.575725 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:10.575814 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:10.577055 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.577771 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.579316 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.579683 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.579772 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.583633 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:10.583646 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:10.583662 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:10.583716 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:10.586486 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:10.587148 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:10.587156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:10.587158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:10.587276 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:10.587578 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:10.587581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:10.587583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:10.587933 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:10.588151 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:10.588465 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 23:25:10.588618 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:25:10.589536 16224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:25:10.677553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 16101
I20260620 23:25:10.694239 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 16234
Jun 20 23:25:10 dist-test-slave-5kzz krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:25:10.708343 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:10.694972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.0.97.3:46745)
0620 23:25:10.695135 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:10.695139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:10.695156 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:25:10.695333 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:10.695337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:10.695413 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:10.695499 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:10.696782 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.697481 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.698918 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:10.699288 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:10.699385 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:10.703339 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:10.703354 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:10.703370 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:10.703401 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:10.705845 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:10.706429 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:10.706436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:10.706438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:10.706569 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:10.706982 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:10.706986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:10.706987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:10.707321 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:10.707529 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:10.707769 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 23:25:10.707974 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:25:10.708969 16357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:25:10.794667 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 16234
I20260620 23:25:10.813185 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 15874
I20260620 23:25:10.814342 15936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:25:10.920639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 15874
I20260620 23:25:30.941576 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42629
--webserver_interface=127.0.97.62
--webserver_port=32987
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:25:31.254422 16411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:25:31.254695 16411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:25:31.254864 16411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:25:31.261950 16411 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:25:31.262033 16411 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:25:31.262112 16411 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:25:31.262187 16411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:25:31.262234 16411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:25:31.262360 16411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:25:31.262409 16411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:25:31.262506 16411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:25:31.272485 16411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=32987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:25:31.273722 16411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:25:31.275053 16411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:25:31.296895 16418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:31.296927 16417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:31.296969 16420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:25:31.297854 16411 server_base.cc:1034] running on GCE node
I20260620 23:25:31.298768 16411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:25:31.300349 16411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:25:31.301582 16411 hybrid_clock.cc:648] HybridClock initialized: now 1781997931301505 us; error 54 us; skew 500 ppm
I20260620 23:25:31.301973 16411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:25:31 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:25:31.308956 16411 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:25:31.310556 16411 webserver.cc:460] Webserver started at http://127.0.97.62:32987/ using document root <none> and password file <none>
I20260620 23:25:31.311403 16411 fs_manager.cc:359] Metadata directory not provided
I20260620 23:25:31.311502 16411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:25:31.317195 16411 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 23:25:31.320016 16427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:31.321295 16411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:25:31.321432 16411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "615ec1f9d0ff4f128604310003c2590e"
format_stamp: "Formatted at 2026-06-20 23:25:08 on dist-test-slave-5kzz"
server_key: "801700cfbc188126c4327238652597aa"
server_key_iv: "161477ce8cc932effb315449e3842128"
server_key_version: "encryptionkey@0"
I20260620 23:25:31.322398 16411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:25:31.345772 16411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:25:31.349793 16411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:25:31.350063 16411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:25:31.368651 16411 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42629
I20260620 23:25:31.368688 16478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42629 every 8 connection(s)
I20260620 23:25:31.370151 16411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:25:31.373315 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 16411
I20260620 23:25:31.374251 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45227
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42629
--webserver_port=35053
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:25:31.377398 16479 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:25:31.380692 16479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Bootstrap starting.
I20260620 23:25:31.399914 16479 log.cc:824] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:25:31.411698 16479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:25:31.412241 16479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Bootstrap complete.
I20260620 23:25:31.421238 16479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:25:31.422046 16479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615ec1f9d0ff4f128604310003c2590e, State: Initialized, Role: FOLLOWER
I20260620 23:25:31.422781 16479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:25:31.423031 16479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:25:31.423149 16479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:25:31.423363 16479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:25:31.427386 16479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:25:31.427894 16479 leader_election.cc:302] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 615ec1f9d0ff4f128604310003c2590e; no voters:
I20260620 23:25:31.428604 16479 leader_election.cc:288] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:25:31.428752 16483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:25:31.429931 16483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 2 LEADER]: Becoming Leader. State: Replica: 615ec1f9d0ff4f128604310003c2590e, State: Running, Role: LEADER
I20260620 23:25:31.430575 16479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:25:31.430498 16483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:25:31.438830 16485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 615ec1f9d0ff4f128604310003c2590e. Latest consensus state: current_term: 2 leader_uuid: "615ec1f9d0ff4f128604310003c2590e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } } }
I20260620 23:25:31.439183 16485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: This master's current role is: LEADER
I20260620 23:25:31.439754 16484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "615ec1f9d0ff4f128604310003c2590e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } } }
I20260620 23:25:31.440088 16484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: This master's current role is: LEADER
I20260620 23:25:31.446992 16498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:25:31.454324 16498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:25:31.455646 16498 catalog_manager.cc:1259] Loaded cluster ID: a76e42edc22b4728b78d13ac41ff1e21
I20260620 23:25:31.455739 16498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:25:31.459604 16498 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:25:31.461005 16498 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Loaded TSK: 0
I20260620 23:25:31.490357 16498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Generated new TSK 1
I20260620 23:25:31.490761 16498 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:25:31.718117 16481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:25:31.718405 16481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:25:31.718554 16481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:25:31.725752 16481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:25:31.725889 16481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:25:31.726089 16481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:25:31.736234 16481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35053
--tserver_master_addrs=127.0.97.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:25:31.737502 16481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:25:31.738855 16481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:25:31.753456 16510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:31.753448 16508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:31.753602 16507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:25:31.754127 16481 server_base.cc:1034] running on GCE node
I20260620 23:25:31.754851 16481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:25:31.756032 16481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:25:31.757282 16481 hybrid_clock.cc:648] HybridClock initialized: now 1781997931757222 us; error 49 us; skew 500 ppm
I20260620 23:25:31.757671 16481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:25:31 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:25:31.761355 16481 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:25:31.762846 16481 webserver.cc:460] Webserver started at http://127.0.97.1:35053/ using document root <none> and password file <none>
I20260620 23:25:31.763509 16481 fs_manager.cc:359] Metadata directory not provided
I20260620 23:25:31.763643 16481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:25:31.769121 16481 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 23:25:31.771798 16517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:31.773124 16481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:25:31.773284 16481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "31bb8d884058485385ecbe841f6a0a72"
format_stamp: "Formatted at 2026-06-20 23:25:09 on dist-test-slave-5kzz"
server_key: "67279e6e0b45fc773131aea6e853f662"
server_key_iv: "0e87925a48228f9dab7d81628d01be5a"
server_key_version: "encryptionkey@0"
I20260620 23:25:31.774202 16481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:25:31.822199 16481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:25:31.826426 16481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:25:31.826807 16481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:25:31.827744 16481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:25:31.829049 16481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:25:31.829169 16481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:31.829296 16481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:25:31.829393 16481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:31.861611 16481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45227
I20260620 23:25:31.861692 16629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45227 every 8 connection(s)
I20260620 23:25:31.863080 16481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:25:31.867774 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 16481
I20260620 23:25:31.868407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40429
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42629
--webserver_port=33553
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:25:31 dist-test-slave-5kzz krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:25:31.887315 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:31.865687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40189 (local address 127.0.97.62:42629)
0620 23:25:31.866221 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:31.866232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:31.866963 (+ 731us) server_negotiation.cc:407] Connection header received
0620 23:25:31.869232 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:31.869254 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:31.869687 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:31.870097 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:31.871449 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:31.873399 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:31.874984 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:31.876038 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:31.876168 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:31.880165 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:31.880172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:31.880184 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:31.880216 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:31.882976 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:31.883661 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:31.883667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:31.883673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:31.883818 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:31.884238 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:31.884246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:31.884248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:31.884657 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:31.884857 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:31.885582 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 23:25:31.886000 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":230,"threads_started":1}
I20260620 23:25:31.887313 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:31.865931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:40189)
0620 23:25:31.866795 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:31.866839 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:31.868652 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:31.870189 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:31.870195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:31.870742 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:25:31.871254 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:31.871271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:31.873638 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:31.873642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:31.874845 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:31.874857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:31.876265 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:31.876275 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:31.876583 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:31.877303 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:25:31.877336 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:25:31.879890 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:25:31.883108 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:31.883116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:31.883129 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:31.883518 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:31.883937 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:31.883941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:31.883945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:31.884120 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:31.884792 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:25:31.884800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:25:31.885120 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 23:25:31.885367 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 23:25:31.885870 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":342,"threads_started":1}
I20260620 23:25:31.890441 16630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42629
I20260620 23:25:31.890775 16630 heartbeater.cc:464] Registering TS with master...
I20260620 23:25:31.891466 16630 heartbeater.cc:511] Master 127.0.97.62:42629 requested a full tablet report, sending...
I20260620 23:25:31.893517 16444 ts_manager.cc:194] Registered new tserver with Master: 31bb8d884058485385ecbe841f6a0a72 (127.0.97.1:45227)
I20260620 23:25:31.895032 16444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40189
W20260620 23:25:32.181442 16634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:25:32.181700 16634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:25:32.181838 16634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:25:32.188814 16634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:25:32.188911 16634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:25:32.189071 16634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:25:32.198223 16634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33553
--tserver_master_addrs=127.0.97.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:25:32.199484 16634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:25:32.200736 16634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:25:32.213383 16640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:32.213672 16642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:32.213718 16639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:25:32.214515 16634 server_base.cc:1034] running on GCE node
I20260620 23:25:32.215196 16634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:25:32.216187 16634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:25:32.217381 16634 hybrid_clock.cc:648] HybridClock initialized: now 1781997932217346 us; error 42 us; skew 500 ppm
I20260620 23:25:32.217690 16634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:25:32 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:25:32.220819 16634 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:25:32.222182 16634 webserver.cc:460] Webserver started at http://127.0.97.2:33553/ using document root <none> and password file <none>
I20260620 23:25:32.222903 16634 fs_manager.cc:359] Metadata directory not provided
I20260620 23:25:32.223068 16634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:25:32.228322 16634 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:25:32.231071 16649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:32.232514 16634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:25:32.232679 16634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "3a66b333277a422093df07050647b342"
format_stamp: "Formatted at 2026-06-20 23:25:09 on dist-test-slave-5kzz"
server_key: "53ff8f437cfedc07a9feacf523447367"
server_key_iv: "222d29b7e51dddce2530adabfc4742df"
server_key_version: "encryptionkey@0"
I20260620 23:25:32.233669 16634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:25:32.246380 16634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:25:32.250612 16634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:25:32.250931 16634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:25:32.251904 16634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:25:32.253188 16634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:25:32.253309 16634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:32.253448 16634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:25:32.253539 16634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:32.286303 16634 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40429
I20260620 23:25:32.286386 16761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40429 every 8 connection(s)
I20260620 23:25:32.287770 16634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:25:32.292102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 16634
I20260620 23:25:32.292874 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46745
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42629
--webserver_port=43255
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:25:32 dist-test-slave-5kzz krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:25:32.308786 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.290276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40529 (local address 127.0.97.62:42629)
0620 23:25:32.290477 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:32.290482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:32.291607 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 23:25:32.293543 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:32.293549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:32.293645 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:32.293755 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:32.295256 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.295954 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.297453 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.297870 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.298021 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.301907 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:32.301915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:32.301923 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:32.301968 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:32.305463 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:32.306240 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:32.306248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:32.306254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:32.306396 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:32.306895 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:32.306903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:32.306910 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:32.307294 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:32.307448 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:32.308102 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 23:25:32.308354 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:25:32.309401 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.290773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:40529)
0620 23:25:32.291455 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:32.291491 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:32.293272 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:32.293984 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:32.293992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:32.294476 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:25:32.295093 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:32.295110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.296109 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.296120 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:32.297306 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:32.297318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.298103 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.298111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:32.298330 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.299013 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:25:32.299046 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:25:32.301671 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:25:32.305688 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:32.305694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:32.305706 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:32.306107 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:32.306526 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:32.306530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:32.306532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:32.306758 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:32.307418 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:25:32.307423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:25:32.307723 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 23:25:32.307948 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 23:25:32.308324 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":270,"threads_started":1}
I20260620 23:25:32.310994 16762 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42629
I20260620 23:25:32.311311 16762 heartbeater.cc:464] Registering TS with master...
I20260620 23:25:32.312085 16762 heartbeater.cc:511] Master 127.0.97.62:42629 requested a full tablet report, sending...
I20260620 23:25:32.313432 16444 ts_manager.cc:194] Registered new tserver with Master: 3a66b333277a422093df07050647b342 (127.0.97.2:40429)
I20260620 23:25:32.314191 16444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40529
W20260620 23:25:32.615564 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:25:32.615877 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:25:32.616037 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:25:32.623168 16765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:25:32.623308 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:25:32.623477 16765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:25:32.633178 16765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43255
--tserver_master_addrs=127.0.97.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:25:32.634481 16765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:25:32.635819 16765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:25:32.648569 16770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:32.648521 16773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:32.648617 16771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:25:32.649968 16765 server_base.cc:1034] running on GCE node
I20260620 23:25:32.650516 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:25:32.651432 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:25:32.652644 16765 hybrid_clock.cc:648] HybridClock initialized: now 1781997932652598 us; error 43 us; skew 500 ppm
I20260620 23:25:32.652935 16765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:25:32 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:25:32.656008 16765 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:25:32.657505 16765 webserver.cc:460] Webserver started at http://127.0.97.3:43255/ using document root <none> and password file <none>
I20260620 23:25:32.658176 16765 fs_manager.cc:359] Metadata directory not provided
I20260620 23:25:32.658329 16765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:25:32.663496 16765 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:25:32.666049 16780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:32.667377 16765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:25:32.667543 16765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "935fe28ae7fb4ef4bababd665351bbb9"
format_stamp: "Formatted at 2026-06-20 23:25:10 on dist-test-slave-5kzz"
server_key: "86fe8ab194fb68f380e5e4ec4164b50f"
server_key_iv: "c6995a01ef20a211140ea043446251e3"
server_key_version: "encryptionkey@0"
I20260620 23:25:32.668493 16765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:25:32.682363 16765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:25:32.686424 16765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:25:32.686736 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:25:32.687675 16765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:25:32.688928 16765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:25:32.689044 16765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:32.689172 16765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:25:32.689257 16765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:32.722617 16765 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46745
I20260620 23:25:32.722661 16892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46745 every 8 connection(s)
I20260620 23:25:32.724418 16765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:25:32.731017 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 16765
Jun 20 23:25:32 dist-test-slave-5kzz krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:25:32.750589 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.732068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.0.97.62:42629)
0620 23:25:32.735012 (+ 2944us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:32.735018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:32.735038 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:25:32.735103 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:32.735110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:32.735191 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:32.737510 (+ 2319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:32.737696 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.738869 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.740933 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.741694 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.741822 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.744979 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:32.744985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:32.744990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:32.745033 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:32.747803 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:32.748451 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:32.748460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:32.748463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:32.748633 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:32.749131 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:32.749134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:32.749136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:32.749423 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:32.749673 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:32.750010 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 23:25:32.750220 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2816,"thread_start_us":2683,"threads_started":1}
I20260620 23:25:32.754212 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.729384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48515 (local address 127.0.97.62:42629)
0620 23:25:32.729645 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:32.729655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:32.735465 (+ 5810us) server_negotiation.cc:407] Connection header received
0620 23:25:32.738434 (+ 2969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:32.738440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:32.738506 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:32.738567 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:32.741508 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.742426 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.744198 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.744603 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.744738 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.748176 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:32.748181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:32.748184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:32.748210 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:32.751419 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:32.752119 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:32.752125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:32.752128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:32.752259 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:32.752685 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:32.752690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:32.752693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:32.753020 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:32.753127 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:32.753825 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 23:25:32.753989 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 23:25:32.755250 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.728670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:48515)
0620 23:25:32.735314 (+ 6644us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:32.735376 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:32.738133 (+ 2757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:32.739131 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:32.739142 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:32.739928 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:25:32.740738 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:32.740761 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.742588 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.742593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:32.744054 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:32.744065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.744827 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.744834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:32.745083 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.745701 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:25:32.745723 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:25:32.747964 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:25:32.751546 (+ 3582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:32.751554 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:32.751571 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:32.751992 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:32.752370 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:32.752374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:32.752376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:32.752556 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:32.753144 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:25:32.753153 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:25:32.753481 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 23:25:32.753714 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 23:25:32.754090 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6304,"thread_start_us":6049,"threads_started":1}
I20260620 23:25:32.756629 16893 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42629
I20260620 23:25:32.756980 16893 heartbeater.cc:464] Registering TS with master...
I20260620 23:25:32.757682 16893 heartbeater.cc:511] Master 127.0.97.62:42629 requested a full tablet report, sending...
I20260620 23:25:32.759079 16444 ts_manager.cc:194] Registered new tserver with Master: 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745)
I20260620 23:25:32.759830 16444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48515
I20260620 23:25:32.765276 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:25:32.771384 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.766103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.0.97.62:42629)
0620 23:25:32.766342 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:32.766347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:32.766585 (+ 238us) server_negotiation.cc:407] Connection header received
0620 23:25:32.766832 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:32.766836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:32.766971 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:32.767092 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:25:32.767468 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.768171 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.769667 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.770074 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.770164 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.770608 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:25:32.770918 (+ 310us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 23:25:32.771189 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60272: token expired
Metrics: {"server-negotiator.queue_time_us":106}
W20260620 23:25:32.771503 16632 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60272: token expired
W20260620 23:25:32.772202 16365 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:42629)
I20260620 23:25:32.785439 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.772646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60276 (local address 127.0.97.62:42629)
0620 23:25:32.772824 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:32.772829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:32.772926 (+ 97us) server_negotiation.cc:407] Connection header received
0620 23:25:32.773109 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:32.773114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:32.773176 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:32.773278 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:32.774706 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.775404 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.776916 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.777410 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.777533 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.780378 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:32.780384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:32.780387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:32.780418 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:32.782902 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:32.783550 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:32.783561 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:32.783565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:32.783680 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:32.784033 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:32.784039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:32.784042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:32.784336 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:32.784590 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:32.784903 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 23:25:32.785113 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:25:32.788755 16444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:25:32.795833 16444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:25:32.818519 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.807420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.0.97.1:45227)
0620 23:25:32.808041 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:32.808050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:32.809654 (+ 1604us) server_negotiation.cc:407] Connection header received
0620 23:25:32.809957 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:32.809965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:32.810216 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:32.810389 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:25:32.812993 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.814198 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.816496 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.817416 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.817571 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.817771 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:32.818020 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 23:25:32.818210 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":335,"threads_started":1}
I20260620 23:25:32.818589 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.807211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45227 (local address 127.0.0.1:57214)
0620 23:25:32.809510 (+ 2299us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:32.809551 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:32.809757 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:32.810793 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:32.810804 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:32.810847 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:25:32.812556 (+ 1709us) spinlock_profiling.cc:243] Waited 1.07 ms on lock 0x61200002cf28. stack: 00007fe9b609066e 00007fe9b6090430 00007fe9b51175ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fe9b60860fd 00007fe9b6086977 00007fe9ca434a7d 00007fe9ca434a0e 00007fe9b71bf7b7 00007fe9bca9403a 00007fe9bcb1bdca 00007fe9bcb19f68 00007fe9bcb4d82f 00007fe9bcb4d58d
0620 23:25:32.812830 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:32.812843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.814397 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.814402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:32.816347 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:32.816358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.817582 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.817587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:32.817763 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.817817 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 23:25:32.817953 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 23:25:32.818134 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2067,"mutex_wait_us":52,"spinlock_wait_cycles":2457984,"thread_start_us":1982,"threads_started":1}
I20260620 23:25:32.819732 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.807440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40429 (local address 127.0.0.1:59988)
0620 23:25:32.809509 (+ 2069us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:32.809544 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:32.809715 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:32.810938 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:32.810946 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:32.811213 (+ 267us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:25:32.811875 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:32.811894 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.813775 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.813780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:32.816374 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:32.816388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.818507 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.818514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:32.819167 (+ 653us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.819199 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:25:32.819289 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:25:32.819420 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1908,"thread_start_us":1652,"threads_started":1}
I20260620 23:25:32.820039 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.808038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59988 (local address 127.0.97.2:40429)
0620 23:25:32.808613 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:32.808622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:32.809751 (+ 1129us) server_negotiation.cc:407] Connection header received
0620 23:25:32.810318 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:32.810330 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:32.810626 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:32.811942 (+ 1316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:25:32.812144 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.813669 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.816564 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.818033 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.818154 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.818335 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:32.819551 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0620 23:25:32.819720 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":610,"threads_started":1}
I20260620 23:25:32.820942 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.809896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46745 (local address 127.0.0.1:53612)
0620 23:25:32.812523 (+ 2627us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:32.812540 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:32.812660 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:32.813210 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:32.813214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:32.813238 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:25:32.813573 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:32.813581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.815453 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.815458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:32.817704 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:32.817720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.820272 (+ 2552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.820277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:32.820393 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.820414 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:25:32.820518 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:25:32.820634 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2488,"thread_start_us":2343,"threads_started":1}
I20260620 23:25:32.821069 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:32.810231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.0.97.3:46745)
0620 23:25:32.811626 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:32.811637 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:32.812652 (+ 1015us) server_negotiation.cc:407] Connection header received
0620 23:25:32.812815 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:32.812822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:32.813043 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:32.813226 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:25:32.813727 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.815288 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.817872 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:32.819783 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:32.819918 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:32.820133 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:32.820568 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 23:25:32.820720 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1222,"thread_start_us":1415,"threads_started":1}
I20260620 23:25:32.823022 16565 tablet_service.cc:1467] Processing CreateTablet for tablet bb47cf77f427422a9f78c4404632ec01 (DEFAULT_TABLE table=test-table [id=603fea6850ba4c8496031f40c9e38cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:25:32.823503 16697 tablet_service.cc:1467] Processing CreateTablet for tablet bb47cf77f427422a9f78c4404632ec01 (DEFAULT_TABLE table=test-table [id=603fea6850ba4c8496031f40c9e38cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:25:32.823992 16828 tablet_service.cc:1467] Processing CreateTablet for tablet bb47cf77f427422a9f78c4404632ec01 (DEFAULT_TABLE table=test-table [id=603fea6850ba4c8496031f40c9e38cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:25:32.824517 16565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb47cf77f427422a9f78c4404632ec01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:25:32.824721 16697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb47cf77f427422a9f78c4404632ec01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:25:32.825176 16828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb47cf77f427422a9f78c4404632ec01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:25:32.838258 16909 tablet_bootstrap.cc:492] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9: Bootstrap starting.
I20260620 23:25:32.840179 16910 tablet_bootstrap.cc:492] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342: Bootstrap starting.
I20260620 23:25:32.841845 16911 tablet_bootstrap.cc:492] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Bootstrap starting.
I20260620 23:25:32.842697 16909 tablet_bootstrap.cc:654] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9: Neither blocks nor log segments found. Creating new log.
I20260620 23:25:32.843637 16910 tablet_bootstrap.cc:654] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342: Neither blocks nor log segments found. Creating new log.
I20260620 23:25:32.844156 16909 log.cc:824] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:25:32.844864 16910 log.cc:824] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342: Log is configured to *not* fsync() on all Append() calls
I20260620 23:25:32.845141 16911 tablet_bootstrap.cc:654] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Neither blocks nor log segments found. Creating new log.
I20260620 23:25:32.846406 16911 log.cc:824] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Log is configured to *not* fsync() on all Append() calls
I20260620 23:25:32.847901 16909 tablet_bootstrap.cc:492] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9: No bootstrap required, opened a new log
I20260620 23:25:32.847903 16910 tablet_bootstrap.cc:492] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342: No bootstrap required, opened a new log
I20260620 23:25:32.848172 16909 ts_tablet_manager.cc:1397] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 23:25:32.848172 16910 ts_tablet_manager.cc:1397] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 23:25:32.848938 16911 tablet_bootstrap.cc:492] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: No bootstrap required, opened a new log
I20260620 23:25:32.849188 16911 ts_tablet_manager.cc:1397] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 23:25:32.856173 16911 raft_consensus.cc:348] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:32.856638 16911 raft_consensus.cc:374] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:25:32.856740 16911 raft_consensus.cc:729] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31bb8d884058485385ecbe841f6a0a72, State: Initialized, Role: FOLLOWER
I20260620 23:25:32.857811 16911 consensus_queue.cc:260] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:32.858983 16630 heartbeater.cc:503] Master 127.0.97.62:42629 was elected leader, sending a full tablet report...
I20260620 23:25:32.858722 16910 raft_consensus.cc:348] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:32.859158 16910 raft_consensus.cc:374] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:25:32.859264 16910 raft_consensus.cc:729] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a66b333277a422093df07050647b342, State: Initialized, Role: FOLLOWER
I20260620 23:25:32.859689 16911 ts_tablet_manager.cc:1428] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260620 23:25:32.859848 16910 consensus_queue.cc:260] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:32.858722 16909 raft_consensus.cc:348] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:32.860344 16909 raft_consensus.cc:374] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:25:32.860468 16909 raft_consensus.cc:729] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935fe28ae7fb4ef4bababd665351bbb9, State: Initialized, Role: FOLLOWER
I20260620 23:25:32.861166 16909 consensus_queue.cc:260] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:32.861699 16910 ts_tablet_manager.cc:1428] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260620 23:25:32.865314 16631 tablet.cc:2369] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:25:32.867435 16893 heartbeater.cc:503] Master 127.0.97.62:42629 was elected leader, sending a full tablet report...
I20260620 23:25:32.867820 16909 ts_tablet_manager.cc:1428] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
I20260620 23:25:32.868389 16762 heartbeater.cc:503] Master 127.0.97.62:42629 was elected leader, sending a full tablet report...
W20260620 23:25:32.991837 16894 tablet.cc:2369] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:25:33.039710 16763 tablet.cc:2369] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:25:33.041558 16915 raft_consensus.cc:482] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:25:33.041887 16915 raft_consensus.cc:504] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:33.043835 16915 leader_election.cc:288] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a66b333277a422093df07050647b342 (127.0.97.2:40429), 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745)
I20260620 23:25:33.053031 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.044320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55029 (local address 127.0.97.2:40429)
0620 23:25:33.044570 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:33.044580 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:33.045249 (+ 669us) server_negotiation.cc:407] Connection header received
0620 23:25:33.045409 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:33.045414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:33.045497 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:33.045599 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:25:33.046459 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.047581 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.050552 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.051794 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.051947 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:33.052065 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:33.052643 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 23:25:33.052782 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:25:33.053037 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.044319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58621 (local address 127.0.97.3:46745)
0620 23:25:33.044651 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:33.044660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:33.045253 (+ 593us) server_negotiation.cc:407] Connection header received
0620 23:25:33.045424 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:33.045428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:33.045497 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:33.045598 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:25:33.046459 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.047582 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.050544 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.051817 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.051954 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:33.052068 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:33.052633 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 23:25:33.052770 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260620 23:25:33.053304 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.044232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40429 (local address 127.0.97.1:55029)
0620 23:25:33.045066 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:33.045085 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:33.045238 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:33.045629 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:33.045634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:33.045663 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:25:33.046260 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:33.046275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.047712 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.047717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:33.050365 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:33.050381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.052093 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.052097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:33.052322 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:33.052465 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 23:25:33.052564 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:25:33.052791 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":335,"thread_start_us":457,"threads_started":1}
I20260620 23:25:33.053326 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.044284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46745 (local address 127.0.97.1:58621)
0620 23:25:33.045125 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:33.045141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:33.045291 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:33.045629 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:33.045634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:33.045672 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:25:33.046297 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:33.046306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.047712 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.047717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:33.050372 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:33.050381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.052078 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.052086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:33.052324 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:33.052465 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 23:25:33.052585 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 23:25:33.052791 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":328,"spinlock_wait_cycles":50304,"thread_start_us":616,"threads_started":1}
I20260620 23:25:33.054580 16717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb47cf77f427422a9f78c4404632ec01" candidate_uuid: "31bb8d884058485385ecbe841f6a0a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a66b333277a422093df07050647b342" is_pre_election: true
I20260620 23:25:33.054580 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb47cf77f427422a9f78c4404632ec01" candidate_uuid: "31bb8d884058485385ecbe841f6a0a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935fe28ae7fb4ef4bababd665351bbb9" is_pre_election: true
I20260620 23:25:33.055171 16717 raft_consensus.cc:2454] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31bb8d884058485385ecbe841f6a0a72 in term 0.
I20260620 23:25:33.055171 16848 raft_consensus.cc:2454] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31bb8d884058485385ecbe841f6a0a72 in term 0.
I20260620 23:25:33.055920 16519 leader_election.cc:302] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 31bb8d884058485385ecbe841f6a0a72, 3a66b333277a422093df07050647b342, 935fe28ae7fb4ef4bababd665351bbb9; no voters:
I20260620 23:25:33.056370 16915 raft_consensus.cc:2781] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:25:33.056535 16915 raft_consensus.cc:482] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:25:33.056645 16915 raft_consensus.cc:3037] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:25:33.059192 16915 raft_consensus.cc:504] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:33.059773 16915 leader_election.cc:288] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 1 election: Requested vote from peers 3a66b333277a422093df07050647b342 (127.0.97.2:40429), 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745)
I20260620 23:25:33.060252 16717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb47cf77f427422a9f78c4404632ec01" candidate_uuid: "31bb8d884058485385ecbe841f6a0a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a66b333277a422093df07050647b342"
I20260620 23:25:33.060258 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb47cf77f427422a9f78c4404632ec01" candidate_uuid: "31bb8d884058485385ecbe841f6a0a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935fe28ae7fb4ef4bababd665351bbb9"
I20260620 23:25:33.060469 16848 raft_consensus.cc:3037] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:25:33.060458 16717 raft_consensus.cc:3037] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:25:33.062598 16848 raft_consensus.cc:2454] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31bb8d884058485385ecbe841f6a0a72 in term 1.
I20260620 23:25:33.062772 16717 raft_consensus.cc:2454] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31bb8d884058485385ecbe841f6a0a72 in term 1.
I20260620 23:25:33.063073 16519 leader_election.cc:302] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31bb8d884058485385ecbe841f6a0a72, 935fe28ae7fb4ef4bababd665351bbb9; no voters:
I20260620 23:25:33.063346 16915 raft_consensus.cc:2781] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:25:33.063786 16915 raft_consensus.cc:686] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 LEADER]: Becoming Leader. State: Replica: 31bb8d884058485385ecbe841f6a0a72, State: Running, Role: LEADER
I20260620 23:25:33.064265 16915 consensus_queue.cc:237] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:33.069037 16444 catalog_manager.cc:5495] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31bb8d884058485385ecbe841f6a0a72 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "31bb8d884058485385ecbe841f6a0a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } health_report { overall_health: UNKNOWN } } }
I20260620 23:25:33.111462 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.106109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.0.97.1:45227)
0620 23:25:33.106293 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:33.106297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:33.106413 (+ 116us) server_negotiation.cc:407] Connection header received
0620 23:25:33.106619 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:33.106623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:33.106787 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:33.106951 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:25:33.107305 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.108022 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.109379 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.109761 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.109850 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:33.110312 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:25:33.110943 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:25:33.111049 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:33.111167 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 23:25:33.111232 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:25:33.119807 16717 raft_consensus.cc:1261] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342 [term 1 FOLLOWER]: Refusing update from remote peer 31bb8d884058485385ecbe841f6a0a72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:25:33.119807 16848 raft_consensus.cc:1261] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9 [term 1 FOLLOWER]: Refusing update from remote peer 31bb8d884058485385ecbe841f6a0a72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:25:33.121035 16920 consensus_queue.cc:1035] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:25:33.121469 16915 consensus_queue.cc:1035] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:25:33.149464 16443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:25:33.149816 16443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:25:33.153712 16443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Sending DeleteTablet for 3 replicas of tablet bb47cf77f427422a9f78c4404632ec01
I20260620 23:25:33.155158 16828 tablet_service.cc:1514] Processing DeleteTablet for tablet bb47cf77f427422a9f78c4404632ec01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:25:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53612
I20260620 23:25:33.155341 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 16481
I20260620 23:25:33.155251 16565 tablet_service.cc:1514] Processing DeleteTablet for tablet bb47cf77f427422a9f78c4404632ec01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:25:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57214
I20260620 23:25:33.156260 16697 tablet_service.cc:1514] Processing DeleteTablet for tablet bb47cf77f427422a9f78c4404632ec01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:25:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59988
I20260620 23:25:33.171286 16934 tablet_replica.cc:330] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342: stopping tablet replica
I20260620 23:25:33.172556 16932 tablet_replica.cc:330] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9: stopping tablet replica
I20260620 23:25:33.172015 16934 raft_consensus.cc:2229] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:25:33.172879 16933 tablet_replica.cc:330] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: stopping tablet replica
I20260620 23:25:33.173444 16933 raft_consensus.cc:2229] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:25:33.173815 16934 raft_consensus.cc:2258] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:25:33.173967 16932 raft_consensus.cc:2229] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:25:33.174172 16933 raft_consensus.cc:2258] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:25:33.174758 16932 raft_consensus.cc:2258] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:25:33.176088 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.156655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.0.97.1:45227)
0620 23:25:33.156870 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:33.156875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:33.156889 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:25:33.156936 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:33.156940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:33.157009 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:33.157132 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:33.159485 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.160377 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.163041 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.163481 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.163636 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:33.168249 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:33.168261 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:33.168276 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:33.168321 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:33.172350 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:33.173286 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:33.173295 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:33.173299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:33.173491 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:33.173977 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:33.173982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:33.173985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:33.174474 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:33.174843 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:33.175468 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 23:25:33.175704 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:25:33.176992 16933 ts_tablet_manager.cc:1905] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:25:33.177174 16625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:25:33.178359 16934 ts_tablet_manager.cc:1905] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:25:33.179920 16932 ts_tablet_manager.cc:1905] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:25:33.184913 16932 ts_tablet_manager.cc:1918] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:25:33.185081 16932 log.cc:1196] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/bb47cf77f427422a9f78c4404632ec01
I20260620 23:25:33.185521 16932 ts_tablet_manager.cc:1939] T bb47cf77f427422a9f78c4404632ec01 P 935fe28ae7fb4ef4bababd665351bbb9: Deleting consensus metadata
I20260620 23:25:33.186066 16934 ts_tablet_manager.cc:1918] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:25:33.186218 16934 log.cc:1196] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/bb47cf77f427422a9f78c4404632ec01
I20260620 23:25:33.186735 16934 ts_tablet_manager.cc:1939] T bb47cf77f427422a9f78c4404632ec01 P 3a66b333277a422093df07050647b342: Deleting consensus metadata
I20260620 23:25:33.187288 16429 catalog_manager.cc:4841] TS 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745): tablet bb47cf77f427422a9f78c4404632ec01 (table test-table [id=603fea6850ba4c8496031f40c9e38cfe]) successfully deleted
I20260620 23:25:33.188447 16430 catalog_manager.cc:4841] TS 3a66b333277a422093df07050647b342 (127.0.97.2:40429): tablet bb47cf77f427422a9f78c4404632ec01 (table test-table [id=603fea6850ba4c8496031f40c9e38cfe]) successfully deleted
I20260620 23:25:33.274807 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 16481
W20260620 23:25:33.293653 16429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45227 (error 108)
I20260620 23:25:33.293805 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 16634
I20260620 23:25:33.296217 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.295746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45227 (local address 127.0.0.1:57258)
0620 23:25:33.295955 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:33.296064 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 23:25:33.296802 16429 catalog_manager.cc:4584] TS 31bb8d884058485385ecbe841f6a0a72 (127.0.97.1:45227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb47cf77f427422a9f78c4404632ec01: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45227: connect: Connection refused (error 111)
I20260620 23:25:33.311470 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.294519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.0.97.2:40429)
0620 23:25:33.294723 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:33.294730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:33.294870 (+ 140us) server_negotiation.cc:407] Connection header received
0620 23:25:33.295084 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:33.295092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:33.295187 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:33.295312 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:33.297338 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.298526 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.299867 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.300504 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.300620 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:33.305988 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:33.305997 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:33.306011 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:33.306040 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:33.308940 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:33.309614 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:33.309621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:33.309623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:33.309734 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:33.310053 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:33.310057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:33.310059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:33.310397 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:33.310619 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:33.310944 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:25:33.311148 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:25:33.312114 16757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:25:33.350658 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.350120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45227 (local address 127.0.0.1:57270)
0620 23:25:33.350367 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:33.350475 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 23:25:33.351764 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.351359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45227 (local address 127.0.0.1:57280)
0620 23:25:33.351508 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:33.351605 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 23:25:33.403712 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 16634
I20260620 23:25:33.412271 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.411800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45227 (local address 127.0.0.1:57290)
0620 23:25:33.412007 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:33.412111 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 23:25:33.413326 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.413005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45227 (local address 127.0.0.1:57292)
0620 23:25:33.413150 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:33.413216 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 23:25:33.420890 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 16765
I20260620 23:25:33.433846 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.421547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.0.97.3:46745)
0620 23:25:33.421737 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:33.421743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:33.421762 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:25:33.421857 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:33.421863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:33.421950 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:33.422056 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:33.423440 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.424127 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.425457 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:33.425908 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:33.426039 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:33.428560 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:33.428573 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:33.428590 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:33.428619 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:33.431305 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:33.432004 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:33.432013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:33.432015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:33.432224 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:33.432571 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:33.432575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:33.432577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:33.432909 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:33.433186 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:33.433388 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 23:25:33.433541 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:25:33.434660 16888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:25:33.494254 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.493663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45227 (local address 127.0.0.1:57298)
0620 23:25:33.493907 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:33.494038 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 23:25:33.495512 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.495072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45227 (local address 127.0.0.1:57304)
0620 23:25:33.495293 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:33.495376 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 23:25:33.545938 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 16765
I20260620 23:25:33.566401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 16411
I20260620 23:25:33.567720 16474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:25:33.682485 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:33.681793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45227 (local address 127.0.0.1:57312)
0620 23:25:33.681986 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:33.682094 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 23:25:33.682561 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 16411
I20260620 23:25:53.701756 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42629
--webserver_interface=127.0.97.62
--webserver_port=32987
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:25:54.036635 16940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:25:54.037029 16940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:25:54.037194 16940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:25:54.044636 16940 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:25:54.044781 16940 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:25:54.044883 16940 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:25:54.044970 16940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:25:54.045068 16940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:25:54.045193 16940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:25:54.045280 16940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:25:54.045360 16940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:25:54.055768 16940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=32987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:25:54.057194 16940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:25:54.058580 16940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:25:54.070926 16945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:54.071062 16946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:54.071062 16948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:25:54.071939 16940 server_base.cc:1034] running on GCE node
I20260620 23:25:54.072849 16940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:25:54.074381 16940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:25:54.075688 16940 hybrid_clock.cc:648] HybridClock initialized: now 1781997954075542 us; error 115 us; skew 500 ppm
I20260620 23:25:54.076167 16940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:25:54 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:25:54.079555 16940 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:25:54.081295 16940 webserver.cc:460] Webserver started at http://127.0.97.62:32987/ using document root <none> and password file <none>
I20260620 23:25:54.082093 16940 fs_manager.cc:359] Metadata directory not provided
I20260620 23:25:54.082242 16940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:25:54.088258 16940 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 23:25:54.091696 16955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:54.093320 16940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:25:54.093526 16940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "615ec1f9d0ff4f128604310003c2590e"
format_stamp: "Formatted at 2026-06-20 23:25:08 on dist-test-slave-5kzz"
server_key: "801700cfbc188126c4327238652597aa"
server_key_iv: "161477ce8cc932effb315449e3842128"
server_key_version: "encryptionkey@0"
I20260620 23:25:54.094542 16940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:25:54.136508 16940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:25:54.140872 16940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:25:54.141186 16940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:25:54.160298 16940 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42629
I20260620 23:25:54.160337 17006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42629 every 8 connection(s)
I20260620 23:25:54.162118 16940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:25:54.168246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 16940
I20260620 23:25:54.169176 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45227
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42629
--webserver_port=35053
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:25:54.170256 17007 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:25:54.173812 17007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Bootstrap starting.
I20260620 23:25:54.203998 17007 log.cc:824] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:25:54.221832 17007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:25:54.222429 17007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Bootstrap complete.
I20260620 23:25:54.231827 17007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:25:54.232795 17007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615ec1f9d0ff4f128604310003c2590e, State: Initialized, Role: FOLLOWER
I20260620 23:25:54.233497 17007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:25:54.233747 17007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:25:54.233891 17007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:25:54.234097 17007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:25:54.238061 17007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:25:54.238601 17007 leader_election.cc:302] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 615ec1f9d0ff4f128604310003c2590e; no voters:
I20260620 23:25:54.239423 17007 leader_election.cc:288] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:25:54.239588 17011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:25:54.240715 17011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [term 3 LEADER]: Becoming Leader. State: Replica: 615ec1f9d0ff4f128604310003c2590e, State: Running, Role: LEADER
I20260620 23:25:54.241377 17011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:25:54.241693 17007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:25:54.243983 17013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 615ec1f9d0ff4f128604310003c2590e. Latest consensus state: current_term: 3 leader_uuid: "615ec1f9d0ff4f128604310003c2590e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } } }
I20260620 23:25:54.244412 17013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: This master's current role is: LEADER
I20260620 23:25:54.245010 17012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "615ec1f9d0ff4f128604310003c2590e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615ec1f9d0ff4f128604310003c2590e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } } }
I20260620 23:25:54.245327 17012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e [sys.catalog]: This master's current role is: LEADER
I20260620 23:25:54.252085 17017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:25:54.263616 17017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:25:54.265710 17017 catalog_manager.cc:1259] Loaded cluster ID: a76e42edc22b4728b78d13ac41ff1e21
I20260620 23:25:54.265913 17017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:25:54.270298 17029 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:25:54.271193 17017 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:25:54.272399 17017 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Loaded TSK: 0
I20260620 23:25:54.272557 17017 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 615ec1f9d0ff4f128604310003c2590e: Loaded TSK: 1
F20260620 23:25:54.273375 17017 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc502c4f9ad google::LogMessage::Fail() at ??:0
@ 0x7fc502c5288a google::LogMessage::SendToLog() at ??:0
@ 0x7fc502c4f4de google::LogMessage::Flush() at ??:0
@ 0x7fc502c50a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc5069b37d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc5069a273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc51867e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc51867d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc5187092a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fc5187090c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fc5187535e9 std::function<>::operator()() at ??:0
@ 0x7fc518681cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fc51867fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc518707610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fc518707439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc50589b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc50589e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fc50589e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc505879eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc508b536db start_thread at ??:0
@ 0x7fc501ae961f clone at ??:0
W20260620 23:25:54.604336 17009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:25:54.604640 17009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:25:54.604740 17009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:25:54.612293 17009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:25:54.612414 17009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:25:54.612529 17009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:25:54.622560 17009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35053
--tserver_master_addrs=127.0.97.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:25:54.624058 17009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:25:54.625391 17009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:25:54.639498 17055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:54.639345 17058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:54.639508 17056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:25:54.640928 17009 server_base.cc:1034] running on GCE node
I20260620 23:25:54.641940 17009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:25:54.643149 17009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:25:54.644392 17009 hybrid_clock.cc:648] HybridClock initialized: now 1781997954644342 us; error 44 us; skew 500 ppm
I20260620 23:25:54.644734 17009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:25:54 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:25:54.648084 17009 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:25:54.649677 17009 webserver.cc:460] Webserver started at http://127.0.97.1:35053/ using document root <none> and password file <none>
I20260620 23:25:54.650411 17009 fs_manager.cc:359] Metadata directory not provided
I20260620 23:25:54.650563 17009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:25:54.655978 17009 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:25:54.659027 17065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:54.660451 17009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:25:54.660646 17009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "31bb8d884058485385ecbe841f6a0a72"
format_stamp: "Formatted at 2026-06-20 23:25:09 on dist-test-slave-5kzz"
server_key: "67279e6e0b45fc773131aea6e853f662"
server_key_iv: "0e87925a48228f9dab7d81628d01be5a"
server_key_version: "encryptionkey@0"
I20260620 23:25:54.661657 17009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:25:54.673169 17009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:25:54.677254 17009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:25:54.677541 17009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:25:54.678531 17009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:25:54.680394 17072 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 23:25:54.686048 17009 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:25:54.686203 17009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 23:25:54.686362 17009 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:25:54.689671 17009 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:25:54.689785 17009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260620 23:25:54.690145 17072 tablet_bootstrap.cc:492] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Bootstrap starting.
I20260620 23:25:54.698961 17072 log.cc:824] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Log is configured to *not* fsync() on all Append() calls
I20260620 23:25:54.719343 17072 tablet_bootstrap.cc:492] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:25:54.720609 17072 tablet_bootstrap.cc:492] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Bootstrap complete.
I20260620 23:25:54.721381 17072 ts_tablet_manager.cc:1397] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.012s
I20260620 23:25:54.736742 17009 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45227
I20260620 23:25:54.736797 17179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45227 every 8 connection(s)
I20260620 23:25:54.736321 17072 raft_consensus.cc:348] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:54.737964 17072 raft_consensus.cc:729] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31bb8d884058485385ecbe841f6a0a72, State: Initialized, Role: FOLLOWER
I20260620 23:25:54.738478 17009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:25:54.738912 17072 consensus_queue.cc:260] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:54.739339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 17009
I20260620 23:25:54.740110 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40429
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42629
--webserver_port=33553
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:25:54.747957 17072 ts_tablet_manager.cc:1428] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72: Time spent starting tablet: real 0.026s user 0.026s sys 0.000s
I20260620 23:25:54.753006 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:54.741652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:51557)
0620 23:25:54.751455 (+ 9803us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:54.751730 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9408,"thread_start_us":9190,"threads_started":1}
W20260620 23:25:54.753868 17180 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42629 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42629: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
I20260620 23:25:54.754896 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:54.754497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:41463)
0620 23:25:54.754664 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:54.754767 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:25:54.755959 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:54.755604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:36175)
0620 23:25:54.755747 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:54.755844 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:25:54.756263 17180 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:25:55.075162 17184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:25:55.075503 17184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:25:55.075666 17184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:25:55.083051 17184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:25:55.083201 17184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:25:55.083379 17184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:25:55.093268 17184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33553
--tserver_master_addrs=127.0.97.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:25:55.094606 17184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:25:55.095960 17184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:25:55.109736 17190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:55.109707 17192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:55.109685 17189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:25:55.110563 17184 server_base.cc:1034] running on GCE node
I20260620 23:25:55.111377 17184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:25:55.112502 17184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:25:55.113705 17184 hybrid_clock.cc:648] HybridClock initialized: now 1781997955113656 us; error 41 us; skew 500 ppm
I20260620 23:25:55.114064 17184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:25:55 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:25:55.117681 17184 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:25:55.119725 17184 webserver.cc:460] Webserver started at http://127.0.97.2:33553/ using document root <none> and password file <none>
I20260620 23:25:55.120474 17184 fs_manager.cc:359] Metadata directory not provided
I20260620 23:25:55.120607 17184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:25:55.126375 17184 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.001s
I20260620 23:25:55.129467 17199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:55.131094 17184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:25:55.131244 17184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "3a66b333277a422093df07050647b342"
format_stamp: "Formatted at 2026-06-20 23:25:09 on dist-test-slave-5kzz"
server_key: "53ff8f437cfedc07a9feacf523447367"
server_key_iv: "222d29b7e51dddce2530adabfc4742df"
server_key_version: "encryptionkey@0"
I20260620 23:25:55.132309 17184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:25:55.153357 17184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:25:55.157938 17184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:25:55.158260 17184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:25:55.159338 17184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:25:55.160742 17184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:25:55.160828 17184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:55.160977 17184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:25:55.161069 17184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:55.195823 17184 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40429
I20260620 23:25:55.195931 17311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40429 every 8 connection(s)
I20260620 23:25:55.197336 17184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:25:55.200727 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 17184
I20260620 23:25:55.201385 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46745
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42629
--webserver_port=43255
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:25:55.202407 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:55.200281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:55117)
0620 23:25:55.201134 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:55.201384 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":344,"threads_started":1}
W20260620 23:25:55.203109 17312 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42629 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42629: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
I20260620 23:25:55.203987 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:55.203639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:58801)
0620 23:25:55.203790 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:55.203884 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 23:25:55.204885 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:55.204564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:44535)
0620 23:25:55.204705 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:55.204785 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:25:55.205181 17312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:25:55.527498 17315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:25:55.527803 17315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:25:55.527961 17315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:25:55.535368 17315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:25:55.535481 17315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:25:55.535652 17315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:25:55.545647 17315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43255
--tserver_master_addrs=127.0.97.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:25:55.546984 17315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:25:55.548280 17315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:25:55.561590 17320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:55.561590 17323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:25:55.561542 17321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:25:55.562372 17315 server_base.cc:1034] running on GCE node
I20260620 23:25:55.563032 17315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:25:55.563957 17315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:25:55.565156 17315 hybrid_clock.cc:648] HybridClock initialized: now 1781997955565090 us; error 53 us; skew 500 ppm
I20260620 23:25:55.565440 17315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:25:55 dist-test-slave-5kzz krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:25:55.568578 17315 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:25:55.570108 17315 webserver.cc:460] Webserver started at http://127.0.97.3:43255/ using document root <none> and password file <none>
I20260620 23:25:55.571069 17315 fs_manager.cc:359] Metadata directory not provided
I20260620 23:25:55.571221 17315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:25:55.579030 17315 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:25:55.581763 17330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:55.583278 17315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:25:55.583423 17315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "935fe28ae7fb4ef4bababd665351bbb9"
format_stamp: "Formatted at 2026-06-20 23:25:10 on dist-test-slave-5kzz"
server_key: "86fe8ab194fb68f380e5e4ec4164b50f"
server_key_iv: "c6995a01ef20a211140ea043446251e3"
server_key_version: "encryptionkey@0"
I20260620 23:25:55.584364 17315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:25:55.606999 17315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:25:55.610978 17315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:25:55.611238 17315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:25:55.612288 17315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:25:55.613608 17315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:25:55.613704 17315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:55.613842 17315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:25:55.613945 17315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:25:55.647964 17315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46745
I20260620 23:25:55.648059 17442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46745 every 8 connection(s)
I20260620 23:25:55.649425 17315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:25:55.654147 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:55.652328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:56547)
0620 23:25:55.653057 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:55.653276 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":283,"threads_started":1}
W20260620 23:25:55.654858 17443 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42629 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42629: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
I20260620 23:25:55.655632 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:55.655315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:33559)
0620 23:25:55.655448 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:55.655518 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 23:25:55.656309 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 17315
I20260620 23:25:55.656592 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:55.656250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:34357)
0620 23:25:55.656385 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:55.656492 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:25:55.656864 17443 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:25:55.757905 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:55.756922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:37165)
0620 23:25:55.757459 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:55.757577 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":276,"threads_started":1}
I20260620 23:25:56.206449 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:56.205670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:45373)
0620 23:25:56.206132 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:56.206219 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260620 23:25:56.308235 17449 raft_consensus.cc:482] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:25:56.308552 17449 raft_consensus.cc:504] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:56.310308 17449 leader_election.cc:288] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a66b333277a422093df07050647b342 (127.0.97.2:40429), 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745)
Jun 20 23:25:56 dist-test-slave-5kzz krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Jun 20 23:25:56 dist-test-slave-5kzz krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:25:56.348747 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:56.311071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33505 (local address 127.0.97.3:46745)
0620 23:25:56.321963 (+ 10892us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:56.321990 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:56.323220 (+ 1230us) server_negotiation.cc:407] Connection header received
0620 23:25:56.326010 (+ 2790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:56.326025 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:56.326594 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:56.327041 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:56.330229 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:56.332190 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:56.335229 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:56.336336 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:56.336539 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:56.341769 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:56.341777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:56.341790 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:56.341827 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:56.344870 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:56.345755 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:56.345762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:56.345767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:56.345918 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:56.346416 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:56.346420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:56.346422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:56.347000 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:56.347188 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:56.347977 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 23:25:56.348326 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10616,"thread_start_us":10456,"threads_started":1}
I20260620 23:25:56.348753 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:56.310986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57079 (local address 127.0.97.2:40429)
0620 23:25:56.322805 (+ 11819us) server_negotiation.cc:206] Beginning negotiation
0620 23:25:56.322823 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:25:56.322879 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:25:56.326038 (+ 3159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:25:56.326048 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:25:56.326594 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:25:56.327041 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:25:56.330232 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:56.332190 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:56.335228 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:25:56.336336 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:25:56.336539 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:56.341542 (+ 5003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:25:56.341557 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:25:56.341583 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:25:56.341652 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:25:56.344866 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:56.345741 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:56.345747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:56.345753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:56.345913 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:25:56.346392 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:25:56.346398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:25:56.346406 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:25:56.347001 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:25:56.347188 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 23:25:56.347966 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 23:25:56.348361 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11548,"thread_start_us":11324,"threads_started":1}
I20260620 23:25:56.348773 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:56.310841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46745 (local address 127.0.97.1:33505)
0620 23:25:56.323070 (+ 12229us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:56.323087 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:56.323415 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:56.327235 (+ 3820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:56.327247 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:56.328639 (+ 1392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:25:56.329966 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:56.329998 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:56.332429 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:56.332434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:56.335056 (+ 2622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:56.335073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:56.336654 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:56.336663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:56.337057 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:56.338042 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:25:56.338087 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:25:56.341496 (+ 3409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:25:56.345038 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:56.345048 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:56.345064 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:56.345586 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:56.346042 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:56.346047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:56.346051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:56.346275 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:56.347147 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:25:56.347154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:25:56.347479 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 23:25:56.347754 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 23:25:56.348311 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12109,"mutex_wait_us":594,"spinlock_wait_cycles":28416,"thread_start_us":11891,"threads_started":1}
I20260620 23:25:56.348909 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:56.310725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40429 (local address 127.0.97.1:57079)
0620 23:25:56.322514 (+ 11789us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:56.322574 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 23:25:56.323421 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:25:56.327235 (+ 3814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:25:56.327247 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:25:56.328639 (+ 1392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:25:56.329966 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:56.329998 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:56.332431 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:56.332435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:56.335055 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:25:56.335073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:25:56.336654 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:25:56.336663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:25:56.337056 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:25:56.338042 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:25:56.338088 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:25:56.341188 (+ 3100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:25:56.345038 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:56.345047 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:56.345064 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:56.345578 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:56.346050 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:25:56.346053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:25:56.346055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:25:56.346268 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:25:56.347148 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:25:56.347155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:25:56.347479 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 23:25:56.347755 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 23:25:56.348486 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11657,"mutex_wait_us":760,"spinlock_wait_cycles":91904,"thread_start_us":11287,"threads_started":1}
W20260620 23:25:56.352788 17068 leader_election.cc:341] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a66b333277a422093df07050647b342 (127.0.97.2:40429): Not found: Tablet not found: bb47cf77f427422a9f78c4404632ec01
W20260620 23:25:56.353705 17067 leader_election.cc:341] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745): Not found: Tablet not found: bb47cf77f427422a9f78c4404632ec01
I20260620 23:25:56.353883 17067 leader_election.cc:302] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31bb8d884058485385ecbe841f6a0a72; no voters: 3a66b333277a422093df07050647b342, 935fe28ae7fb4ef4bababd665351bbb9
I20260620 23:25:56.354319 17449 raft_consensus.cc:2726] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:25:56.658244 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:56.657391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:34343)
0620 23:25:56.657868 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:56.657991 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":244,"threads_started":1}
I20260620 23:25:56.759465 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:56.759021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:47193)
0620 23:25:56.759214 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:56.759319 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:25:57.208362 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:57.207482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:57647)
0620 23:25:57.208102 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:57.208196 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":448,"threads_started":1}
I20260620 23:25:57.659996 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:57.659308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:57269)
0620 23:25:57.659758 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:57.659836 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":231,"threads_started":1}
I20260620 23:25:57.761336 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:57.760519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:40505)
0620 23:25:57.761082 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:57.761160 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":247,"threads_started":1}
I20260620 23:25:58.210014 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:58.209313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:37731)
0620 23:25:58.209759 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:58.209837 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":246,"threads_started":1}
I20260620 23:25:58.298810 17459 raft_consensus.cc:482] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:25:58.299044 17459 raft_consensus.cc:504] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:25:58.299723 17459 leader_election.cc:288] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a66b333277a422093df07050647b342 (127.0.97.2:40429), 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745)
W20260620 23:25:58.300647 17068 leader_election.cc:341] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a66b333277a422093df07050647b342 (127.0.97.2:40429): Not found: Tablet not found: bb47cf77f427422a9f78c4404632ec01
W20260620 23:25:58.300850 17067 leader_election.cc:341] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745): Not found: Tablet not found: bb47cf77f427422a9f78c4404632ec01
I20260620 23:25:58.300980 17067 leader_election.cc:302] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31bb8d884058485385ecbe841f6a0a72; no voters: 3a66b333277a422093df07050647b342, 935fe28ae7fb4ef4bababd665351bbb9
I20260620 23:25:58.301252 17459 raft_consensus.cc:2726] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:25:58.661791 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:58.660981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:46279)
0620 23:25:58.661520 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:58.661614 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":278,"threads_started":1}
I20260620 23:25:58.763234 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:58.762350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:35845)
0620 23:25:58.762929 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:58.763032 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":229,"threads_started":1}
I20260620 23:25:59.211838 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:59.211075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:58413)
0620 23:25:59.211565 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:59.211653 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":209,"threads_started":1}
I20260620 23:25:59.663878 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:59.662869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:41045)
0620 23:25:59.663572 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:59.663681 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":466,"threads_started":1}
I20260620 23:25:59.765421 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:25:59.764479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:34959)
0620 23:25:59.765089 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 23:25:59.765181 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":257,"threads_started":1}
I20260620 23:26:00.213539 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:00.212808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:47841)
0620 23:26:00.213255 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:00.213341 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":226,"threads_started":1}
I20260620 23:26:00.665815 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:00.664948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:50559)
0620 23:26:00.665509 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:00.665593 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":301,"threads_started":1}
I20260620 23:26:00.767267 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:00.766471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:44767)
0620 23:26:00.767020 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:00.767110 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":325,"threads_started":1}
I20260620 23:26:01.215667 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:01.214749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:59549)
0620 23:26:01.215373 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:01.215477 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":317,"threads_started":1}
I20260620 23:26:01.426628 17469 raft_consensus.cc:482] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:26:01.427000 17469 raft_consensus.cc:504] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:26:01.427695 17469 leader_election.cc:288] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a66b333277a422093df07050647b342 (127.0.97.2:40429), 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745)
W20260620 23:26:01.428856 17068 leader_election.cc:341] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a66b333277a422093df07050647b342 (127.0.97.2:40429): Not found: Tablet not found: bb47cf77f427422a9f78c4404632ec01
W20260620 23:26:01.429026 17067 leader_election.cc:341] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745): Not found: Tablet not found: bb47cf77f427422a9f78c4404632ec01
I20260620 23:26:01.429176 17067 leader_election.cc:302] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31bb8d884058485385ecbe841f6a0a72; no voters: 3a66b333277a422093df07050647b342, 935fe28ae7fb4ef4bababd665351bbb9
I20260620 23:26:01.429469 17469 raft_consensus.cc:2726] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:26:01.667671 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:01.666895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:48613)
0620 23:26:01.667400 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:01.667490 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":253,"threads_started":1}
I20260620 23:26:01.769133 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:01.768207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:48677)
0620 23:26:01.768819 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:01.768925 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":307,"threads_started":1}
I20260620 23:26:02.217629 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:02.216825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:37237)
0620 23:26:02.217363 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:02.217442 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":219,"threads_started":1}
I20260620 23:26:02.669494 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:02.668729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:35569)
0620 23:26:02.669210 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:02.669307 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":218,"threads_started":1}
I20260620 23:26:02.770911 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:02.770096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:39525)
0620 23:26:02.770567 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:02.770741 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":255,"threads_started":1}
I20260620 23:26:03.219647 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:03.218881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:42499)
0620 23:26:03.219366 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:03.219470 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":227,"threads_started":1}
I20260620 23:26:03.671293 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:03.670530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:50143)
0620 23:26:03.671020 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:03.671105 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":279,"threads_started":1}
I20260620 23:26:03.772763 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:03.771924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:59967)
0620 23:26:03.772437 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:03.772536 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":262,"threads_started":1}
I20260620 23:26:04.221513 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:04.220698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:35821)
0620 23:26:04.221242 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:04.221348 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":261,"threads_started":1}
I20260620 23:26:04.673009 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:04.672222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:35417)
0620 23:26:04.672757 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:04.672846 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":244,"threads_started":1}
I20260620 23:26:04.774806 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:04.773954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:59637)
0620 23:26:04.774446 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:04.774568 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":260,"threads_started":1}
I20260620 23:26:05.223347 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:05.222556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:40715)
0620 23:26:05.223074 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:05.223161 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":275,"threads_started":1}
I20260620 23:26:05.675088 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:05.674236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:44575)
0620 23:26:05.674776 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:05.674866 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":249,"threads_started":1}
I20260620 23:26:05.776762 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:05.775921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:54209)
0620 23:26:05.776484 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:05.776573 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":332,"threads_started":1}
I20260620 23:26:06.225337 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:06.224391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:45059)
0620 23:26:06.225075 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:06.225167 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":269,"threads_started":1}
I20260620 23:26:06.274938 17485 raft_consensus.cc:482] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:26:06.275206 17485 raft_consensus.cc:504] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:26:06.275950 17485 leader_election.cc:288] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a66b333277a422093df07050647b342 (127.0.97.2:40429), 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745)
W20260620 23:26:06.277034 17068 leader_election.cc:341] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a66b333277a422093df07050647b342 (127.0.97.2:40429): Not found: Tablet not found: bb47cf77f427422a9f78c4404632ec01
W20260620 23:26:06.277249 17067 leader_election.cc:341] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745): Not found: Tablet not found: bb47cf77f427422a9f78c4404632ec01
I20260620 23:26:06.277359 17067 leader_election.cc:302] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31bb8d884058485385ecbe841f6a0a72; no voters: 3a66b333277a422093df07050647b342, 935fe28ae7fb4ef4bababd665351bbb9
I20260620 23:26:06.277586 17485 raft_consensus.cc:2726] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:26:06.676906 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:06.676141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:48401)
0620 23:26:06.676635 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:06.676737 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":232,"threads_started":1}
I20260620 23:26:06.778658 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:06.777780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:42003)
0620 23:26:06.778406 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:06.778483 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":408,"threads_started":1}
I20260620 23:26:07.227349 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:07.226489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:56965)
0620 23:26:07.227088 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:07.227173 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":369,"threads_started":1}
I20260620 23:26:07.678642 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:07.677868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:49831)
0620 23:26:07.678385 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:07.678462 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":289,"threads_started":1}
I20260620 23:26:07.780344 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:07.779611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:33185)
0620 23:26:07.780075 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:07.780164 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":244,"threads_started":1}
I20260620 23:26:08.229285 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:08.228442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:39877)
0620 23:26:08.229026 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:08.229122 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":310,"threads_started":1}
I20260620 23:26:08.680541 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:08.679670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:49259)
0620 23:26:08.680289 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:08.680373 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":291,"threads_started":1}
I20260620 23:26:08.782109 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:08.781379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:45879)
0620 23:26:08.781845 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:08.781926 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":234,"threads_started":1}
I20260620 23:26:09.231088 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:09.230245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:50461)
0620 23:26:09.230802 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:09.230887 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":256,"threads_started":1}
I20260620 23:26:09.393680 17495 raft_consensus.cc:482] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:26:09.393924 17495 raft_consensus.cc:504] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bb8d884058485385ecbe841f6a0a72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45227 } } peers { permanent_uuid: "3a66b333277a422093df07050647b342" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40429 } } peers { permanent_uuid: "935fe28ae7fb4ef4bababd665351bbb9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46745 } }
I20260620 23:26:09.394577 17495 leader_election.cc:288] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a66b333277a422093df07050647b342 (127.0.97.2:40429), 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745)
W20260620 23:26:09.395644 17067 leader_election.cc:341] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 935fe28ae7fb4ef4bababd665351bbb9 (127.0.97.3:46745): Not found: Tablet not found: bb47cf77f427422a9f78c4404632ec01
W20260620 23:26:09.395916 17068 leader_election.cc:341] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a66b333277a422093df07050647b342 (127.0.97.2:40429): Not found: Tablet not found: bb47cf77f427422a9f78c4404632ec01
I20260620 23:26:09.396121 17068 leader_election.cc:302] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31bb8d884058485385ecbe841f6a0a72; no voters: 3a66b333277a422093df07050647b342, 935fe28ae7fb4ef4bababd665351bbb9
I20260620 23:26:09.396378 17495 raft_consensus.cc:2726] T bb47cf77f427422a9f78c4404632ec01 P 31bb8d884058485385ecbe841f6a0a72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:26:09.682251 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:09.681466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:42059)
0620 23:26:09.681977 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:09.682059 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":272,"threads_started":1}
I20260620 23:26:09.784250 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:09.783447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.1:39763)
0620 23:26:09.783934 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:09.784025 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":239,"threads_started":1}
I20260620 23:26:10.233166 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:10.232419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.2:44259)
0620 23:26:10.232905 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:10.233006 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":235,"threads_started":1}
I20260620 23:26:10.665388 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 17009
I20260620 23:26:10.684278 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:10.683497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.97.3:48279)
0620 23:26:10.684000 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:10.684097 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":258,"threads_started":1}
I20260620 23:26:10.687088 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:10.667033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.0.97.1:45227)
0620 23:26:10.667655 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:10.667667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:10.667709 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:26:10.667789 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:10.667798 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:10.668080 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:10.668223 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:10.669649 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:10.670742 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:10.671989 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:10.672593 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:10.672699 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:10.680732 (+ 8033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:10.680747 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:10.680764 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:10.680812 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:10.684279 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:10.685073 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:10.685080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:10.685082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:10.685211 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:10.685617 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:10.685621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:10.685623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:10.686007 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:10.686237 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:10.686555 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 23:26:10.686837 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":321,"threads_started":1}
I20260620 23:26:10.688510 17175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:10.779858 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 17009
I20260620 23:26:10.798494 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 17184
I20260620 23:26:10.812090 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:10.799381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.0.97.2:40429)
0620 23:26:10.799933 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:10.799937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:10.799954 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:26:10.800007 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:10.800010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:10.800068 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:10.800162 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:10.801505 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:10.802384 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:10.803793 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:10.804230 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:10.804343 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:10.806851 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:10.806856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:10.806859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:10.806894 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:10.809632 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:10.810263 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:10.810267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:10.810269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:10.810407 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:10.810759 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:10.810763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:10.810765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:10.811040 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:10.811236 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:10.811487 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 23:26:10.811655 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":320,"threads_started":1}
I20260620 23:26:10.812819 17307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:10.898437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 17184
I20260620 23:26:10.916453 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 17315
I20260620 23:26:10.930059 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:10.917349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.0.97.3:46745)
0620 23:26:10.917936 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:10.917941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:10.917958 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:26:10.918015 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:10.918019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:10.918090 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:10.918188 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:10.919704 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:10.920518 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:10.921717 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:10.922146 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:10.922246 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:10.924928 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:10.924933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:10.924937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:10.924968 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:10.927654 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:10.928276 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:10.928279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:10.928281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:10.928421 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:10.928736 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:10.928739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:10.928741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:10.928979 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:10.929171 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:10.929561 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 23:26:10.929748 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":319,"threads_started":1}
I20260620 23:26:10.930925 17438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:11.016243 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 17315
2026-06-20T23:26:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64153 ms)
[----------] 30 tests from SecurityITest (290442 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:11 dist-test-slave-5kzz krb5kdc[17508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:11 dist-test-slave-5kzz krb5kdc[17508](info): set up 2 sockets
Jun 20 23:26:11 dist-test-slave-5kzz krb5kdc[17508](info): commencing operation
krb5kdc: starting...
W20260620 23:26:12.126912 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:12 dist-test-slave-5kzz krb5kdc[17508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:26:12.390817 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36957
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:12.729332 17521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:12.729660 17521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:12.729807 17521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:12.737177 17521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:12.737298 17521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:12.737388 17521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:12.737486 17521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:12.737568 17521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:12.747659 17521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:12.749068 17521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:12.750430 17521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:26:12.763126 17521 server_base.cc:1034] running on GCE node
W20260620 23:26:12.763126 17526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:12.763428 17529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:12.763069 17527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:12.764429 17521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:12.765873 17521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:12.767161 17521 hybrid_clock.cc:648] HybridClock initialized: now 1781997972767082 us; error 53 us; skew 500 ppm
I20260620 23:26:12.767591 17521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:12 dist-test-slave-5kzz krb5kdc[17508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:12.770987 17521 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:26:12.772712 17521 webserver.cc:460] Webserver started at http://127.0.97.62:35981/ using document root <none> and password file <none>
I20260620 23:26:12.773466 17521 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:12.773622 17521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:12.773944 17521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:12.777073 17521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "4cd9c322afbb47c79f3a36f1be7301a7"
format_stamp: "Formatted at 2026-06-20 23:26:12 on dist-test-slave-5kzz"
server_key: "a636500bfa08bf8c5ac0bc0b8b046038"
server_key_iv: "c732095e44f2fd1c39fea65d54340fda"
server_key_version: "encryptionkey@0"
I20260620 23:26:12.777897 17521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "4cd9c322afbb47c79f3a36f1be7301a7"
format_stamp: "Formatted at 2026-06-20 23:26:12 on dist-test-slave-5kzz"
server_key: "a636500bfa08bf8c5ac0bc0b8b046038"
server_key_iv: "c732095e44f2fd1c39fea65d54340fda"
server_key_version: "encryptionkey@0"
I20260620 23:26:12.783588 17521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:26:12.787443 17536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:12.788945 17521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 23:26:12.789160 17521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "4cd9c322afbb47c79f3a36f1be7301a7"
format_stamp: "Formatted at 2026-06-20 23:26:12 on dist-test-slave-5kzz"
server_key: "a636500bfa08bf8c5ac0bc0b8b046038"
server_key_iv: "c732095e44f2fd1c39fea65d54340fda"
server_key_version: "encryptionkey@0"
I20260620 23:26:12.789408 17521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:12.823659 17521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:12.828231 17521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:12.828536 17521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:12.848265 17521 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36957
I20260620 23:26:12.848318 17587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36957 every 8 connection(s)
I20260620 23:26:12.849824 17521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:12.853901 17588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:12.856424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 17521
I20260620 23:26:12.856662 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:12.857292 388 external_mini_cluster.cc:1428] Setting key 8c1c7a21d02295a670ea9621a12e4a12
I20260620 23:26:12.867148 17588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7: Bootstrap starting.
I20260620 23:26:12.871182 17588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:12.872658 17588 log.cc:824] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:26:12 dist-test-slave-5kzz krb5kdc[17508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:12.876121 17588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7: No bootstrap required, opened a new log
I20260620 23:26:12.885710 17591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:12.859506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.0.97.62:36957)
0620 23:26:12.860316 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:12.860333 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:12.860384 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:26:12.862850 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:12.862887 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:12.863470 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:12.863942 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:12.865295 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:12.867314 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:12.868962 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:12.869935 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:12.870080 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:12.875285 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:12.875298 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:12.875319 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:12.875369 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:12.879670 (+ 4301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:12.880658 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:12.880675 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:12.880687 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:12.880983 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:12.881612 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:12.881621 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:12.881626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:12.882489 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:12.882865 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:12.883698 (+ 833us) server_negotiation.cc:299] Negotiation successful
0620 23:26:12.884207 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":341,"threads_started":1}
I20260620 23:26:12.887813 17588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd9c322afbb47c79f3a36f1be7301a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36957 } }
I20260620 23:26:12.888218 17588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:12.888326 17588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd9c322afbb47c79f3a36f1be7301a7, State: Initialized, Role: FOLLOWER
I20260620 23:26:12.888950 17588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd9c322afbb47c79f3a36f1be7301a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36957 } }
I20260620 23:26:12.889235 17588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:12.889396 17588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:12.889580 17588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:12.891808 17588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd9c322afbb47c79f3a36f1be7301a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36957 } }
I20260620 23:26:12.892343 17588 leader_election.cc:302] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd9c322afbb47c79f3a36f1be7301a7; no voters:
I20260620 23:26:12.893019 17588 leader_election.cc:288] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:12.893154 17593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:12.894563 17593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd9c322afbb47c79f3a36f1be7301a7, State: Running, Role: LEADER
I20260620 23:26:12.895186 17593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd9c322afbb47c79f3a36f1be7301a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36957 } }
I20260620 23:26:12.895339 17588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:12.897833 17595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cd9c322afbb47c79f3a36f1be7301a7. Latest consensus state: current_term: 1 leader_uuid: "4cd9c322afbb47c79f3a36f1be7301a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd9c322afbb47c79f3a36f1be7301a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36957 } } }
I20260620 23:26:12.897914 17594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cd9c322afbb47c79f3a36f1be7301a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd9c322afbb47c79f3a36f1be7301a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36957 } } }
I20260620 23:26:12.898267 17594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:12.898273 17595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:12.899232 17601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:12.905048 17601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:12.914018 17601 catalog_manager.cc:1347] Generated new cluster ID: 5b2bfd596ec5432788007507096f3688
I20260620 23:26:12.914141 17601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:12.934151 17601 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:12.935606 17601 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:12.946656 17601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7: Generated new TSK 0
I20260620 23:26:12.947554 17601 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:26:13.011190 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36957
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:13.349932 17614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:13.350286 17614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:13.350466 17614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:13.358078 17614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:13.358242 17614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:13.358419 17614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:26:13.368455 17614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:13.369791 17614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:13.371249 17614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:13.385424 17619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:13.385622 17622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:13.385509 17620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:13.388343 17614 server_base.cc:1034] running on GCE node
I20260620 23:26:13.389104 17614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:13.390244 17614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:13.391532 17614 hybrid_clock.cc:648] HybridClock initialized: now 1781997973391461 us; error 65 us; skew 500 ppm
I20260620 23:26:13.391937 17614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:13 dist-test-slave-5kzz krb5kdc[17508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:13.395466 17614 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:26:13.397352 17614 webserver.cc:460] Webserver started at http://127.0.97.1:36577/ using document root <none> and password file <none>
I20260620 23:26:13.398068 17614 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:13.398172 17614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:13.398428 17614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:13.401546 17614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "56e7169f2b0940ccba822eeb34d04ffa"
format_stamp: "Formatted at 2026-06-20 23:26:13 on dist-test-slave-5kzz"
server_key: "1d606804ebd6101657ef80dd0e87af9a"
server_key_iv: "d089a9856634c175a0eab74e0671379d"
server_key_version: "encryptionkey@0"
I20260620 23:26:13.402307 17614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "56e7169f2b0940ccba822eeb34d04ffa"
format_stamp: "Formatted at 2026-06-20 23:26:13 on dist-test-slave-5kzz"
server_key: "1d606804ebd6101657ef80dd0e87af9a"
server_key_iv: "d089a9856634c175a0eab74e0671379d"
server_key_version: "encryptionkey@0"
I20260620 23:26:13.407634 17614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 23:26:13.411216 17629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:13.412739 17614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:26:13.412892 17614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "56e7169f2b0940ccba822eeb34d04ffa"
format_stamp: "Formatted at 2026-06-20 23:26:13 on dist-test-slave-5kzz"
server_key: "1d606804ebd6101657ef80dd0e87af9a"
server_key_iv: "d089a9856634c175a0eab74e0671379d"
server_key_version: "encryptionkey@0"
I20260620 23:26:13.413082 17614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:13.431458 17614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:13.436255 17614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:13.436517 17614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:13.437527 17614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:13.439019 17614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:13.439100 17614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:13.439208 17614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:13.439258 17614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:13.474141 17614 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38957
I20260620 23:26:13.474184 17741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38957 every 8 connection(s)
I20260620 23:26:13.475497 17614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:26:13.478992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 17614
I20260620 23:26:13.479262 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:26:13.480019 388 external_mini_cluster.cc:1428] Setting key 374a422ec1fc3a3c7dc5aaf724ad85b0
Jun 20 23:26:13 dist-test-slave-5kzz krb5kdc[17508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:13.495752 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:13.478084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34055 (local address 127.0.97.62:36957)
0620 23:26:13.478579 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:13.478583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:13.479725 (+ 1142us) server_negotiation.cc:407] Connection header received
0620 23:26:13.481653 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:13.481659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:13.481751 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:13.481866 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:13.483452 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:13.484158 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:13.485725 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:13.486128 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:13.486262 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:13.489718 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:13.489723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:13.489727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:13.489767 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:13.492404 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:13.493271 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:13.493284 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:13.493297 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:13.493412 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:13.493895 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:13.493898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:13.493900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:13.494215 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:13.494368 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:13.495147 (+ 779us) server_negotiation.cc:299] Negotiation successful
0620 23:26:13.495373 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":281,"threads_started":1}
I20260620 23:26:13.496747 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:13.478633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36957 (local address 127.0.97.1:34055)
0620 23:26:13.479552 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:13.479598 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:13.481378 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:13.482106 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:13.482114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:13.482647 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:13.483280 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:13.483297 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:13.484330 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:13.484335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:13.485578 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:13.485589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:13.486369 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:13.486375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:13.486608 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:13.487366 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:26:13.487396 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:13.489497 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:13.492570 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:13.492578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:13.492595 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:13.493149 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:13.493531 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:13.493535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:13.493539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:13.493729 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:13.494339 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:13.494347 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:13.494745 (+ 398us) client_negotiation.cc:769] Sending connection context
0620 23:26:13.495045 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 23:26:13.495579 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":355,"threads_started":1}
I20260620 23:26:13.498555 17742 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36957
I20260620 23:26:13.498946 17742 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:13.499976 17742 heartbeater.cc:511] Master 127.0.97.62:36957 requested a full tablet report, sending...
I20260620 23:26:13.502761 17553 ts_manager.cc:194] Registered new tserver with Master: 56e7169f2b0940ccba822eeb34d04ffa (127.0.97.1:38957)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:26:13.504493 17553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:26:13.534788 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36957
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:13.880419 17748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:13.880715 17748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:13.880880 17748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:13.889633 17748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:13.889750 17748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:13.889952 17748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:26:13.900386 17748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:13.901710 17748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:13.903120 17748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:13.916350 17754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:13.916463 17753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:13.916429 17756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:13.917500 17748 server_base.cc:1034] running on GCE node
I20260620 23:26:13.918255 17748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:13.919281 17748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:13.920528 17748 hybrid_clock.cc:648] HybridClock initialized: now 1781997973920460 us; error 55 us; skew 500 ppm
I20260620 23:26:13.920868 17748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:13 dist-test-slave-5kzz krb5kdc[17508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:13.924055 17748 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:26:13.925434 17748 webserver.cc:460] Webserver started at http://127.0.97.2:35823/ using document root <none> and password file <none>
I20260620 23:26:13.926076 17748 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:13.926172 17748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:13.926440 17748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:13.929416 17748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "e99c26d778f044b997ba1c22b504dcc1"
format_stamp: "Formatted at 2026-06-20 23:26:13 on dist-test-slave-5kzz"
server_key: "9a311572575f709813dbfbe9047a71b8"
server_key_iv: "76a17cdd513c6e9e137320c2302dfbe2"
server_key_version: "encryptionkey@0"
I20260620 23:26:13.930198 17748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e99c26d778f044b997ba1c22b504dcc1"
format_stamp: "Formatted at 2026-06-20 23:26:13 on dist-test-slave-5kzz"
server_key: "9a311572575f709813dbfbe9047a71b8"
server_key_iv: "76a17cdd513c6e9e137320c2302dfbe2"
server_key_version: "encryptionkey@0"
I20260620 23:26:13.935572 17748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:26:13.939517 17763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:13.940968 17748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:26:13.941156 17748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "e99c26d778f044b997ba1c22b504dcc1"
format_stamp: "Formatted at 2026-06-20 23:26:13 on dist-test-slave-5kzz"
server_key: "9a311572575f709813dbfbe9047a71b8"
server_key_iv: "76a17cdd513c6e9e137320c2302dfbe2"
server_key_version: "encryptionkey@0"
I20260620 23:26:13.941366 17748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:13.977955 17748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:13.982658 17748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:13.983045 17748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:13.984090 17748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:13.985531 17748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:13.985607 17748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:13.985754 17748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:13.985805 17748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:14.020905 17748 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39337
I20260620 23:26:14.020964 17875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39337 every 8 connection(s)
I20260620 23:26:14.022236 17748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:26:14.029497 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 17748
I20260620 23:26:14.029686 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:26:14.030227 388 external_mini_cluster.cc:1428] Setting key b01b3f587d755ab239f1d1c32e505b92
Jun 20 23:26:14 dist-test-slave-5kzz krb5kdc[17508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:14.043334 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.024853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41867 (local address 127.0.97.62:36957)
0620 23:26:14.025302 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:14.025308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:14.026308 (+ 1000us) server_negotiation.cc:407] Connection header received
0620 23:26:14.028164 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:14.028168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:14.028250 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:14.028349 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:14.029978 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.030990 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.032558 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.032980 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.033117 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.036706 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:14.036712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:14.036715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:14.036749 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:14.039905 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:14.040786 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:14.040794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:14.040798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:14.040943 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:14.041412 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:14.041417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:14.041421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:14.041710 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:14.041877 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:14.042645 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 23:26:14.042919 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":203,"threads_started":1}
I20260620 23:26:14.044404 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.025450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36957 (local address 127.0.97.2:41867)
0620 23:26:14.026132 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:14.026172 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:14.027898 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:14.028575 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:14.028584 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:14.029114 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:14.029788 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.029812 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.031147 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.031153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.032406 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.032419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.033217 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.033224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.033481 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.034192 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:26:14.034222 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:14.036452 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:14.040103 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:14.040111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:14.040128 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:14.040617 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:14.041064 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:14.041069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:14.041073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:14.041284 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:14.041848 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:14.041856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:14.042255 (+ 399us) client_negotiation.cc:769] Sending connection context
0620 23:26:14.042551 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 23:26:14.043167 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":274,"threads_started":1}
I20260620 23:26:14.046027 17876 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36957
I20260620 23:26:14.046401 17876 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:14.047359 17876 heartbeater.cc:511] Master 127.0.97.62:36957 requested a full tablet report, sending...
I20260620 23:26:14.049146 17553 ts_manager.cc:194] Registered new tserver with Master: e99c26d778f044b997ba1c22b504dcc1 (127.0.97.2:39337)
I20260620 23:26:14.050446 17553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41867
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:26:14.086460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36957
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:14.421772 17882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:14.422082 17882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:14.422252 17882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:14.429678 17882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:14.429802 17882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:14.429991 17882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:26:14.440204 17882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:14.441529 17882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:14.442956 17882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:14.456856 17888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:14.456972 17890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:14.456915 17887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:14.457659 17882 server_base.cc:1034] running on GCE node
I20260620 23:26:14.458295 17882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:14.459323 17882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:14.460529 17882 hybrid_clock.cc:648] HybridClock initialized: now 1781997974460484 us; error 35 us; skew 500 ppm
I20260620 23:26:14.460891 17882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:14 dist-test-slave-5kzz krb5kdc[17508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:14.464272 17882 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:26:14.465855 17882 webserver.cc:460] Webserver started at http://127.0.97.3:44385/ using document root <none> and password file <none>
I20260620 23:26:14.466569 17882 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:14.466763 17882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:14.467088 17882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:14.470184 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "b700c7edf6ea42fb9bf06267a5fc3e62"
format_stamp: "Formatted at 2026-06-20 23:26:14 on dist-test-slave-5kzz"
server_key: "c168d29e735906a26763ea6a65815f29"
server_key_iv: "4b8a98090e0910dfca4337b55e559223"
server_key_version: "encryptionkey@0"
I20260620 23:26:14.471056 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b700c7edf6ea42fb9bf06267a5fc3e62"
format_stamp: "Formatted at 2026-06-20 23:26:14 on dist-test-slave-5kzz"
server_key: "c168d29e735906a26763ea6a65815f29"
server_key_iv: "4b8a98090e0910dfca4337b55e559223"
server_key_version: "encryptionkey@0"
I20260620 23:26:14.476586 17882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:26:14.480263 17897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:14.481704 17882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 23:26:14.481881 17882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "b700c7edf6ea42fb9bf06267a5fc3e62"
format_stamp: "Formatted at 2026-06-20 23:26:14 on dist-test-slave-5kzz"
server_key: "c168d29e735906a26763ea6a65815f29"
server_key_iv: "4b8a98090e0910dfca4337b55e559223"
server_key_version: "encryptionkey@0"
I20260620 23:26:14.482096 17882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:14.493947 17882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:14.498574 17882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:14.498932 17882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:14.500001 17882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:14.501452 17882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:14.501557 17882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:14.501718 17882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:14.501793 17882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:14.507849 17742 heartbeater.cc:503] Master 127.0.97.62:36957 was elected leader, sending a full tablet report...
I20260620 23:26:14.537106 17882 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38749
I20260620 23:26:14.537153 18009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38749 every 8 connection(s)
I20260620 23:26:14.538420 17882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:26:14.541484 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 17882
I20260620 23:26:14.541673 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:26:14.542303 388 external_mini_cluster.cc:1428] Setting key eb42f8b459732c884d49c0404fab7503
Jun 20 23:26:14 dist-test-slave-5kzz krb5kdc[17508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:14.559015 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.541104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48145 (local address 127.0.97.62:36957)
0620 23:26:14.541292 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:14.541297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:14.542801 (+ 1504us) server_negotiation.cc:407] Connection header received
0620 23:26:14.544912 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:14.544917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:14.545028 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:14.545137 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:14.546966 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.547699 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.549274 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.549689 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.549830 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.553418 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:14.553423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:14.553427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:14.553461 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:14.555895 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:14.556639 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:14.556649 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:14.556651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:14.556774 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:14.557204 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:14.557211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:14.557218 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:14.557506 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:14.557667 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:14.558426 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 23:26:14.558660 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:26:14.559988 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.541771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36957 (local address 127.0.97.3:48145)
0620 23:26:14.542593 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:14.542631 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:14.544643 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:14.545501 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:14.545521 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:14.546127 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:14.546799 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.546821 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.547854 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.547859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.549102 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.549118 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.549923 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.549930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.550172 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.550950 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:26:14.550973 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:14.553145 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:14.556044 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:14.556054 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:14.556073 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:14.556508 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:14.556881 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:14.556887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:14.556891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:14.557077 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:14.557649 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:14.557657 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:14.558019 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 23:26:14.558302 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 23:26:14.558877 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":319,"threads_started":1}
I20260620 23:26:14.561326 18010 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36957
I20260620 23:26:14.561600 18010 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:14.562243 18010 heartbeater.cc:511] Master 127.0.97.62:36957 requested a full tablet report, sending...
I20260620 23:26:14.563714 17553 ts_manager.cc:194] Registered new tserver with Master: b700c7edf6ea42fb9bf06267a5fc3e62 (127.0.97.3:38749)
I20260620 23:26:14.564568 17553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48145
I20260620 23:26:14.570298 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:26:14.588074 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.574906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.0.97.62:36957)
0620 23:26:14.575128 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:14.575135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:14.575647 (+ 512us) server_negotiation.cc:407] Connection header received
0620 23:26:14.575866 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:14.575871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:14.575935 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:14.576065 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:14.577453 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.578217 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.579603 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.580015 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.580156 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.582557 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:14.582562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:14.582565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:14.582597 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:14.585157 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:14.585893 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:14.585900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:14.585904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:14.586055 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:14.586427 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:14.586433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:14.586436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:14.586879 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:14.587163 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:14.587499 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 23:26:14.587725 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 23:26:14.592280 17553 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:26:14.599740 17553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:26:14.635478 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.612922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.0.97.2:39337)
0620 23:26:14.625869 (+ 12947us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:14.625882 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:14.626521 (+ 639us) server_negotiation.cc:407] Connection header received
0620 23:26:14.626621 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:14.626627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:14.626897 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:14.627938 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:14.628189 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.629875 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.632692 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.634011 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.634133 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.634364 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:14.634992 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 23:26:14.635162 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12796,"thread_start_us":12696,"threads_started":1}
I20260620 23:26:14.635566 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.612696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39337 (local address 127.0.0.1:56420)
0620 23:26:14.626232 (+ 13536us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:14.626252 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:14.626395 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:14.627171 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:14.627181 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:14.627219 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:14.627680 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.627698 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.630045 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.630051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.632524 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.632537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.634505 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.634513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.634755 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.634794 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:26:14.634928 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 23:26:14.635129 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13411,"mutex_wait_us":80,"thread_start_us":11598,"threads_started":1}
I20260620 23:26:14.636834 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.612658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38749 (local address 127.0.0.1:60828)
0620 23:26:14.624875 (+ 12217us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:14.624936 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:14.625247 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:14.627604 (+ 2357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:14.627611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:14.627645 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:14.628086 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.628098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.630045 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.630051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.633740 (+ 3689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.633755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.636021 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.636027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.636192 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.636224 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:26:14.636310 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 23:26:14.636444 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11831,"mutex_wait_us":32,"thread_start_us":14702,"threads_started":1}
I20260620 23:26:14.638036 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.612856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.0.97.3:38749)
0620 23:26:14.626403 (+ 13547us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:14.626411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:14.626462 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:26:14.626582 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:14.626593 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:14.626897 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:14.627105 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:14.628313 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.629890 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.633943 (+ 4053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.635454 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.637168 (+ 1714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.637408 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:14.637503 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 23:26:14.637681 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13363,"thread_start_us":13197,"threads_started":1}
I20260620 23:26:14.638828 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.626006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.0.97.1:38957)
0620 23:26:14.629856 (+ 3850us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:14.629867 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:14.630413 (+ 546us) server_negotiation.cc:407] Connection header received
0620 23:26:14.630517 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:14.630523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:14.630805 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:14.631002 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:14.631590 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.633290 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.635851 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.637346 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.637474 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.637682 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:14.638312 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 23:26:14.638483 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3665,"thread_start_us":3490,"threads_started":1}
I20260620 23:26:14.639590 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.626505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38957 (local address 127.0.0.1:33090)
0620 23:26:14.630157 (+ 3652us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:14.630178 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:14.630331 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:14.630996 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:14.631000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:14.631025 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:14.631421 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.631430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.633455 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.633460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.635676 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.635694 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.637827 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.637833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.638016 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.638053 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:26:14.639084 (+ 1031us) client_negotiation.cc:240] Negotiation successful
0620 23:26:14.639234 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3528,"mutex_wait_us":1,"thread_start_us":3371,"threads_started":1}
I20260620 23:26:14.639453 17811 tablet_service.cc:1467] Processing CreateTablet for tablet 6e160a6c72bb4c4c84c9a6f1ecce780c (DEFAULT_TABLE table=test-table [id=b186683592f94353abe80bdc9f7cf6c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:14.640908 17811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e160a6c72bb4c4c84c9a6f1ecce780c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:14.641796 17945 tablet_service.cc:1467] Processing CreateTablet for tablet 6e160a6c72bb4c4c84c9a6f1ecce780c (DEFAULT_TABLE table=test-table [id=b186683592f94353abe80bdc9f7cf6c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:14.643188 17945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e160a6c72bb4c4c84c9a6f1ecce780c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:14.643198 17677 tablet_service.cc:1467] Processing CreateTablet for tablet 6e160a6c72bb4c4c84c9a6f1ecce780c (DEFAULT_TABLE table=test-table [id=b186683592f94353abe80bdc9f7cf6c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:14.644549 17677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e160a6c72bb4c4c84c9a6f1ecce780c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:14.655169 18028 tablet_bootstrap.cc:492] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62: Bootstrap starting.
I20260620 23:26:14.657254 18029 tablet_bootstrap.cc:492] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1: Bootstrap starting.
I20260620 23:26:14.659096 18028 tablet_bootstrap.cc:654] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:14.659278 18030 tablet_bootstrap.cc:492] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa: Bootstrap starting.
I20260620 23:26:14.660815 18029 tablet_bootstrap.cc:654] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:14.661105 18028 log.cc:824] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:14.662082 18029 log.cc:824] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:14.662663 18030 tablet_bootstrap.cc:654] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:14.663919 18030 log.cc:824] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:14.664450 18028 tablet_bootstrap.cc:492] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62: No bootstrap required, opened a new log
I20260620 23:26:14.664752 18028 ts_tablet_manager.cc:1397] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 23:26:14.665369 18029 tablet_bootstrap.cc:492] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1: No bootstrap required, opened a new log
I20260620 23:26:14.665634 18029 ts_tablet_manager.cc:1397] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:26:14.666555 18030 tablet_bootstrap.cc:492] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa: No bootstrap required, opened a new log
I20260620 23:26:14.666836 18030 ts_tablet_manager.cc:1397] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260620 23:26:14.674623 18030 raft_consensus.cc:348] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38749 } } peers { permanent_uuid: "e99c26d778f044b997ba1c22b504dcc1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39337 } } peers { permanent_uuid: "56e7169f2b0940ccba822eeb34d04ffa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38957 } }
I20260620 23:26:14.675098 18030 raft_consensus.cc:374] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:14.675201 18030 raft_consensus.cc:729] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e7169f2b0940ccba822eeb34d04ffa, State: Initialized, Role: FOLLOWER
I20260620 23:26:14.675871 18030 consensus_queue.cc:260] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38749 } } peers { permanent_uuid: "e99c26d778f044b997ba1c22b504dcc1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39337 } } peers { permanent_uuid: "56e7169f2b0940ccba822eeb34d04ffa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38957 } }
I20260620 23:26:14.676025 18028 raft_consensus.cc:348] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38749 } } peers { permanent_uuid: "e99c26d778f044b997ba1c22b504dcc1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39337 } } peers { permanent_uuid: "56e7169f2b0940ccba822eeb34d04ffa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38957 } }
I20260620 23:26:14.676184 18029 raft_consensus.cc:348] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38749 } } peers { permanent_uuid: "e99c26d778f044b997ba1c22b504dcc1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39337 } } peers { permanent_uuid: "56e7169f2b0940ccba822eeb34d04ffa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38957 } }
I20260620 23:26:14.676491 18028 raft_consensus.cc:374] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:14.676579 18029 raft_consensus.cc:374] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:14.676595 18028 raft_consensus.cc:729] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b700c7edf6ea42fb9bf06267a5fc3e62, State: Initialized, Role: FOLLOWER
I20260620 23:26:14.676689 18029 raft_consensus.cc:729] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99c26d778f044b997ba1c22b504dcc1, State: Initialized, Role: FOLLOWER
I20260620 23:26:14.677412 18028 consensus_queue.cc:260] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38749 } } peers { permanent_uuid: "e99c26d778f044b997ba1c22b504dcc1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39337 } } peers { permanent_uuid: "56e7169f2b0940ccba822eeb34d04ffa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38957 } }
I20260620 23:26:14.677896 18030 ts_tablet_manager.cc:1428] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260620 23:26:14.678573 18010 heartbeater.cc:503] Master 127.0.97.62:36957 was elected leader, sending a full tablet report...
I20260620 23:26:14.678668 18029 consensus_queue.cc:260] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38749 } } peers { permanent_uuid: "e99c26d778f044b997ba1c22b504dcc1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39337 } } peers { permanent_uuid: "56e7169f2b0940ccba822eeb34d04ffa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38957 } }
I20260620 23:26:14.679291 18028 ts_tablet_manager.cc:1428] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260620 23:26:14.680256 17876 heartbeater.cc:503] Master 127.0.97.62:36957 was elected leader, sending a full tablet report...
I20260620 23:26:14.680893 18029 ts_tablet_manager.cc:1428] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260620 23:26:14.727957 17743 tablet.cc:2369] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:26:14.774466 17877 tablet.cc:2369] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:26:14.790131 18011 tablet.cc:2369] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:26:14.835126 18035 raft_consensus.cc:482] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:26:14.835475 18035 raft_consensus.cc:504] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38749 } } peers { permanent_uuid: "e99c26d778f044b997ba1c22b504dcc1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39337 } } peers { permanent_uuid: "56e7169f2b0940ccba822eeb34d04ffa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38957 } }
I20260620 23:26:14.837190 18035 leader_election.cc:288] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e99c26d778f044b997ba1c22b504dcc1 (127.0.97.2:39337), 56e7169f2b0940ccba822eeb34d04ffa (127.0.97.1:38957)
I20260620 23:26:14.844133 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.837817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52661 (local address 127.0.97.2:39337)
0620 23:26:14.838021 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:14.838029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:14.838065 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:26:14.838194 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:14.838202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:14.838296 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:14.838413 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:14.838973 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.839984 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.841868 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.843139 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.843239 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.843313 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:14.843820 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 23:26:14.843932 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:26:14.844203 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.837579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39337 (local address 127.0.97.3:52661)
0620 23:26:14.837803 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:14.837825 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:14.837974 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:14.838474 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:14.838480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:14.838518 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:14.838777 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.838807 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.840136 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.840142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.841713 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.841725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.843386 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.843395 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.843626 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.843663 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:26:14.843768 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:26:14.843911 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 23:26:14.845175 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.837969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38957 (local address 127.0.97.3:39887)
0620 23:26:14.838663 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:14.838731 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:14.838844 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:14.839169 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:14.839174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:14.839199 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:14.839555 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.839566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.840969 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.840973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.842853 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:14.842868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.844575 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.844583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:14.844737 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.844761 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:26:14.844842 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:26:14.844932 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":498,"threads_started":1}
I20260620 23:26:14.845317 17831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e160a6c72bb4c4c84c9a6f1ecce780c" candidate_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99c26d778f044b997ba1c22b504dcc1" is_pre_election: true
I20260620 23:26:14.845855 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.838247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39887 (local address 127.0.97.1:38957)
0620 23:26:14.838468 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:14.838474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:14.838827 (+ 353us) server_negotiation.cc:407] Connection header received
0620 23:26:14.838993 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:14.838997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:14.839055 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:14.839125 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:14.839748 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.840800 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.843010 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.844305 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.844441 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.844559 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:14.845464 (+ 905us) server_negotiation.cc:299] Negotiation successful
0620 23:26:14.845610 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:26:14.845927 17831 raft_consensus.cc:2454] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b700c7edf6ea42fb9bf06267a5fc3e62 in term 0.
I20260620 23:26:14.846715 17900 leader_election.cc:302] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b700c7edf6ea42fb9bf06267a5fc3e62, e99c26d778f044b997ba1c22b504dcc1; no voters:
I20260620 23:26:14.847144 18035 raft_consensus.cc:2781] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:26:14.846978 17697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e160a6c72bb4c4c84c9a6f1ecce780c" candidate_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e7169f2b0940ccba822eeb34d04ffa" is_pre_election: true
I20260620 23:26:14.847318 18035 raft_consensus.cc:482] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:26:14.847425 18035 raft_consensus.cc:3037] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:14.847609 17697 raft_consensus.cc:2454] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b700c7edf6ea42fb9bf06267a5fc3e62 in term 0.
I20260620 23:26:14.850732 18035 raft_consensus.cc:504] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38749 } } peers { permanent_uuid: "e99c26d778f044b997ba1c22b504dcc1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39337 } } peers { permanent_uuid: "56e7169f2b0940ccba822eeb34d04ffa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38957 } }
I20260620 23:26:14.851430 18035 leader_election.cc:288] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [CANDIDATE]: Term 1 election: Requested vote from peers e99c26d778f044b997ba1c22b504dcc1 (127.0.97.2:39337), 56e7169f2b0940ccba822eeb34d04ffa (127.0.97.1:38957)
I20260620 23:26:14.851972 17831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e160a6c72bb4c4c84c9a6f1ecce780c" candidate_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99c26d778f044b997ba1c22b504dcc1"
I20260620 23:26:14.852010 17697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e160a6c72bb4c4c84c9a6f1ecce780c" candidate_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e7169f2b0940ccba822eeb34d04ffa"
I20260620 23:26:14.852209 17697 raft_consensus.cc:3037] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:14.852209 17831 raft_consensus.cc:3037] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:14.855497 17831 raft_consensus.cc:2454] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b700c7edf6ea42fb9bf06267a5fc3e62 in term 1.
I20260620 23:26:14.855497 17697 raft_consensus.cc:2454] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b700c7edf6ea42fb9bf06267a5fc3e62 in term 1.
I20260620 23:26:14.855921 17900 leader_election.cc:302] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e7169f2b0940ccba822eeb34d04ffa, b700c7edf6ea42fb9bf06267a5fc3e62; no voters:
I20260620 23:26:14.856225 18035 raft_consensus.cc:2781] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:14.856662 18035 raft_consensus.cc:686] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 1 LEADER]: Becoming Leader. State: Replica: b700c7edf6ea42fb9bf06267a5fc3e62, State: Running, Role: LEADER
I20260620 23:26:14.857581 18035 consensus_queue.cc:237] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38749 } } peers { permanent_uuid: "e99c26d778f044b997ba1c22b504dcc1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39337 } } peers { permanent_uuid: "56e7169f2b0940ccba822eeb34d04ffa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38957 } }
I20260620 23:26:14.862912 17553 catalog_manager.cc:5495] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 reported cstate change: term changed from 0 to 1, leader changed from <none> to b700c7edf6ea42fb9bf06267a5fc3e62 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b700c7edf6ea42fb9bf06267a5fc3e62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e99c26d778f044b997ba1c22b504dcc1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e7169f2b0940ccba822eeb34d04ffa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38957 } health_report { overall_health: UNKNOWN } } }
I20260620 23:26:14.881873 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.876045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.0.97.3:38749)
0620 23:26:14.876257 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:14.876262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:14.876283 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:26:14.876451 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:14.876456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:14.876534 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:14.876714 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:26:14.877084 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.877813 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.879582 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.880034 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.880236 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.880765 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:26:14.881285 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:26:14.881455 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:14.881510 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 23:26:14.881599 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:26:14.892449 17697 raft_consensus.cc:1261] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa [term 1 FOLLOWER]: Refusing update from remote peer b700c7edf6ea42fb9bf06267a5fc3e62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:26:14.892449 17831 raft_consensus.cc:1261] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1 [term 1 FOLLOWER]: Refusing update from remote peer b700c7edf6ea42fb9bf06267a5fc3e62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:26:14.893644 18038 consensus_queue.cc:1035] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e7169f2b0940ccba822eeb34d04ffa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:26:14.894060 18035 consensus_queue.cc:1035] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e99c26d778f044b997ba1c22b504dcc1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:26:14.924618 17553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:26:14.924957 17553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:26:14.928767 17553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cd9c322afbb47c79f3a36f1be7301a7: Sending DeleteTablet for 3 replicas of tablet 6e160a6c72bb4c4c84c9a6f1ecce780c
I20260620 23:26:14.929960 17945 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e160a6c72bb4c4c84c9a6f1ecce780c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60828
I20260620 23:26:14.930173 17677 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e160a6c72bb4c4c84c9a6f1ecce780c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33090
I20260620 23:26:14.930411 17811 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e160a6c72bb4c4c84c9a6f1ecce780c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56420
I20260620 23:26:14.931336 18050 tablet_replica.cc:330] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62: stopping tablet replica
I20260620 23:26:14.931829 18050 raft_consensus.cc:2229] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:26:14.932036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 17614
I20260620 23:26:14.932578 18050 raft_consensus.cc:2258] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:14.936729 18050 ts_tablet_manager.cc:1905] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:14.944700 18050 ts_tablet_manager.cc:1918] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:14.944934 18050 log.cc:1196] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/6e160a6c72bb4c4c84c9a6f1ecce780c
Jun 20 23:26:14 dist-test-slave-5kzz krb5kdc[17508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:26:14.945590 18050 ts_tablet_manager.cc:1939] T 6e160a6c72bb4c4c84c9a6f1ecce780c P b700c7edf6ea42fb9bf06267a5fc3e62: Deleting consensus metadata
I20260620 23:26:14.947450 18052 tablet_replica.cc:330] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1: stopping tablet replica
I20260620 23:26:14.948066 18052 raft_consensus.cc:2229] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:26:14.948112 17540 catalog_manager.cc:4841] TS b700c7edf6ea42fb9bf06267a5fc3e62 (127.0.97.3:38749): tablet 6e160a6c72bb4c4c84c9a6f1ecce780c (table test-table [id=b186683592f94353abe80bdc9f7cf6c7]) successfully deleted
I20260620 23:26:14.948746 18052 raft_consensus.cc:2258] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:14.949625 18051 tablet_replica.cc:330] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa: stopping tablet replica
I20260620 23:26:14.950088 18051 raft_consensus.cc:2229] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:26:14.950662 18051 raft_consensus.cc:2258] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:14.951754 18052 ts_tablet_manager.cc:1905] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:14.952791 18051 ts_tablet_manager.cc:1905] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:14.956789 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:14.933651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.0.97.1:38957)
0620 23:26:14.933836 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:14.933844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:14.934388 (+ 544us) server_negotiation.cc:407] Connection header received
0620 23:26:14.934661 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:14.934668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:14.934799 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:14.934882 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:14.936623 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.937776 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.939791 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:14.940505 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:14.940621 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:14.946718 (+ 6097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:14.946742 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:14.946762 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:14.946823 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:14.951899 (+ 5076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:14.953115 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:14.953127 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:14.953132 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:14.953343 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:14.953851 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:14.953857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:14.953862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:14.954353 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:14.954722 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:14.956119 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0620 23:26:14.956357 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:26:14.957899 17737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:14.958927 18052 ts_tablet_manager.cc:1918] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:14.959136 18052 log.cc:1196] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/6e160a6c72bb4c4c84c9a6f1ecce780c
I20260620 23:26:14.959746 18052 ts_tablet_manager.cc:1939] T 6e160a6c72bb4c4c84c9a6f1ecce780c P e99c26d778f044b997ba1c22b504dcc1: Deleting consensus metadata
I20260620 23:26:14.962108 17539 catalog_manager.cc:4841] TS e99c26d778f044b997ba1c22b504dcc1 (127.0.97.2:39337): tablet 6e160a6c72bb4c4c84c9a6f1ecce780c (table test-table [id=b186683592f94353abe80bdc9f7cf6c7]) successfully deleted
I20260620 23:26:15.062750 18051 ts_tablet_manager.cc:1918] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:15.062973 18051 log.cc:1196] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/6e160a6c72bb4c4c84c9a6f1ecce780c
I20260620 23:26:15.063534 18051 ts_tablet_manager.cc:1939] T 6e160a6c72bb4c4c84c9a6f1ecce780c P 56e7169f2b0940ccba822eeb34d04ffa: Deleting consensus metadata
I20260620 23:26:15.065534 17539 catalog_manager.cc:4841] TS 56e7169f2b0940ccba822eeb34d04ffa (127.0.97.1:38957): tablet 6e160a6c72bb4c4c84c9a6f1ecce780c (table test-table [id=b186683592f94353abe80bdc9f7cf6c7]) successfully deleted
I20260620 23:26:15.065697 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 17614
I20260620 23:26:15.089636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 17748
Jun 20 23:26:15 dist-test-slave-5kzz krb5kdc[17508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:26:15.105888 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:15.090576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.0.97.2:39337)
0620 23:26:15.090821 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:15.090828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:15.090849 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:26:15.090954 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:15.090960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:15.091049 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:15.091160 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:15.092425 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:15.093324 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:15.094749 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:15.095219 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:15.095354 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:15.100058 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:15.100074 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:15.100090 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:15.100125 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:15.103060 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:15.103808 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:15.103816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:15.103818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:15.103936 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:15.104331 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:15.104335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:15.104337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:15.104713 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:15.105007 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:15.105325 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 23:26:15.105547 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 23:26:15.106647 17871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:15.195201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 17748
I20260620 23:26:15.219151 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 17882
Jun 20 23:26:15 dist-test-slave-5kzz krb5kdc[17508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:26:15.238281 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:15.220038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.0.97.3:38749)
0620 23:26:15.220215 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:15.220220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:15.220238 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:26:15.220370 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:15.220374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:15.220456 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:15.220549 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:15.222338 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:15.223351 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:15.225249 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:15.225716 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:15.225834 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:15.231587 (+ 5753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:15.231603 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:15.231620 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:15.231667 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:15.235142 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:15.235954 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:15.235962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:15.235964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:15.236160 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:15.236584 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:15.236592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:15.236599 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:15.236972 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:15.237325 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:15.237610 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 23:26:15.237859 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:26:15.239035 18005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:15.362030 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 17882
I20260620 23:26:15.382573 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 17521
I20260620 23:26:15.383809 17583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:15.496320 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 17521
2026-06-20T23:26:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4483 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T23:26:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:15Z Disabled control of system clock
I20260620 23:26:15.582425 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42139
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:26:15.917337 18061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:15.917649 18061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:15.917732 18061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:15.925189 18061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:15.925307 18061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:15.925344 18061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:15.925395 18061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:15.925443 18061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:15.934840 18061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:15.936226 18061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:15.937949 18061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:15.950160 18069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:15.950116 18066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:15.950206 18067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:15.951740 18061 server_base.cc:1034] running on GCE node
I20260620 23:26:15.952666 18061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:15.954195 18061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:15.955461 18061 hybrid_clock.cc:648] HybridClock initialized: now 1781997975955417 us; error 60 us; skew 500 ppm
I20260620 23:26:15.955873 18061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:26:15.961879 18061 webserver.cc:460] Webserver started at http://127.0.97.62:44621/ using document root <none> and password file <none>
I20260620 23:26:15.962656 18061 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:15.962868 18061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:15.963213 18061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:15.966315 18061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "495872a5c0a0478baaf4dd96e488016d"
format_stamp: "Formatted at 2026-06-20 23:26:15 on dist-test-slave-5kzz"
server_key: "f567da31af6fa283ae93110cc3abb9b3"
server_key_iv: "681100803dfe247a655b95ed80f6cc5c"
server_key_version: "encryptionkey@0"
I20260620 23:26:15.967219 18061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "495872a5c0a0478baaf4dd96e488016d"
format_stamp: "Formatted at 2026-06-20 23:26:15 on dist-test-slave-5kzz"
server_key: "f567da31af6fa283ae93110cc3abb9b3"
server_key_iv: "681100803dfe247a655b95ed80f6cc5c"
server_key_version: "encryptionkey@0"
I20260620 23:26:15.972645 18061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:26:15.976617 18075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:15.978235 18061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:26:15.978544 18061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "495872a5c0a0478baaf4dd96e488016d"
format_stamp: "Formatted at 2026-06-20 23:26:15 on dist-test-slave-5kzz"
server_key: "f567da31af6fa283ae93110cc3abb9b3"
server_key_iv: "681100803dfe247a655b95ed80f6cc5c"
server_key_version: "encryptionkey@0"
I20260620 23:26:15.978876 18061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:15.998806 18061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:15.999788 18061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:16.000137 18061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:16.020910 18061 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42139
I20260620 23:26:16.020931 18126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42139 every 8 connection(s)
I20260620 23:26:16.022468 18061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:16.026396 18127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:16.027676 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 18061
I20260620 23:26:16.027869 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:16.028472 388 external_mini_cluster.cc:1428] Setting key df4df01b854588a984b93b26e9819399
I20260620 23:26:16.039914 18127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d: Bootstrap starting.
I20260620 23:26:16.042163 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:16.030582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.0.97.62:42139)
0620 23:26:16.031700 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:16.031711 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:16.031758 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:26:16.033445 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:16.033467 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:16.033963 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:16.034386 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:16.034998 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:16.036257 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:16.037704 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:16.038397 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:16.038615 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:16.039812 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:16.039819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:16.039834 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:16.040451 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:16.040531 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:16.040552 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:16.040691 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:16.040908 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 23:26:16.041220 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":357,"threads_started":1}
I20260620 23:26:16.043401 18127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:16.044463 18127 log.cc:824] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:16.047459 18127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d: No bootstrap required, opened a new log
I20260620 23:26:16.055748 18127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495872a5c0a0478baaf4dd96e488016d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42139 } }
I20260620 23:26:16.056202 18127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:16.056329 18127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495872a5c0a0478baaf4dd96e488016d, State: Initialized, Role: FOLLOWER
I20260620 23:26:16.056985 18127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495872a5c0a0478baaf4dd96e488016d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42139 } }
I20260620 23:26:16.057240 18127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:16.057401 18127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:16.057577 18127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:16.059919 18127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495872a5c0a0478baaf4dd96e488016d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42139 } }
I20260620 23:26:16.060513 18127 leader_election.cc:302] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 495872a5c0a0478baaf4dd96e488016d; no voters:
I20260620 23:26:16.061254 18127 leader_election.cc:288] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:16.061416 18132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:16.062201 18132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [term 1 LEADER]: Becoming Leader. State: Replica: 495872a5c0a0478baaf4dd96e488016d, State: Running, Role: LEADER
I20260620 23:26:16.062803 18132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495872a5c0a0478baaf4dd96e488016d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42139 } }
I20260620 23:26:16.063607 18127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:16.065902 18134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 495872a5c0a0478baaf4dd96e488016d. Latest consensus state: current_term: 1 leader_uuid: "495872a5c0a0478baaf4dd96e488016d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495872a5c0a0478baaf4dd96e488016d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42139 } } }
I20260620 23:26:16.065900 18133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "495872a5c0a0478baaf4dd96e488016d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495872a5c0a0478baaf4dd96e488016d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42139 } } }
I20260620 23:26:16.066377 18134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:16.066425 18133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:16.068307 18139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:16.074091 18139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:16.083460 18139 catalog_manager.cc:1347] Generated new cluster ID: bd3178c719e64a0ea970130ee0200adf
I20260620 23:26:16.083580 18139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:16.113972 18139 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:16.115934 18139 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:16.124542 18139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d: Generated new TSK 0
I20260620 23:26:16.125674 18139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:16.140602 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42139
--builtin_ntp_servers=127.0.97.20:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:26:16.503387 18151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:16.503852 18151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:16.504030 18151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:16.511750 18151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:16.511927 18151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:16.512104 18151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:26:16.521641 18151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:16.523068 18151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:16.524956 18151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:16.539186 18156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:16.539276 18157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:16.539237 18159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:16.540227 18151 server_base.cc:1034] running on GCE node
I20260620 23:26:16.540930 18151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:16.541991 18151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:16.543286 18151 hybrid_clock.cc:648] HybridClock initialized: now 1781997976543209 us; error 63 us; skew 500 ppm
I20260620 23:26:16.543763 18151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:26:16.546625 18151 webserver.cc:460] Webserver started at http://127.0.97.1:33019/ using document root <none> and password file <none>
I20260620 23:26:16.547626 18151 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:16.547801 18151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:16.548221 18151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:16.551920 18151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "9e7aec743fba4dfd833819a2f5fa06ba"
format_stamp: "Formatted at 2026-06-20 23:26:16 on dist-test-slave-5kzz"
server_key: "dd85f02ecd44ffb625eaf56c156b932a"
server_key_iv: "dfa321c0088ff908d2abd38570efa0b1"
server_key_version: "encryptionkey@0"
I20260620 23:26:16.552986 18151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9e7aec743fba4dfd833819a2f5fa06ba"
format_stamp: "Formatted at 2026-06-20 23:26:16 on dist-test-slave-5kzz"
server_key: "dd85f02ecd44ffb625eaf56c156b932a"
server_key_iv: "dfa321c0088ff908d2abd38570efa0b1"
server_key_version: "encryptionkey@0"
I20260620 23:26:16.559638 18151 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 23:26:16.563979 18165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:16.565762 18151 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:26:16.566009 18151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "9e7aec743fba4dfd833819a2f5fa06ba"
format_stamp: "Formatted at 2026-06-20 23:26:16 on dist-test-slave-5kzz"
server_key: "dd85f02ecd44ffb625eaf56c156b932a"
server_key_iv: "dfa321c0088ff908d2abd38570efa0b1"
server_key_version: "encryptionkey@0"
I20260620 23:26:16.566289 18151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:16.595503 18151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:16.596472 18151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:16.596787 18151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:16.597859 18151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:16.599537 18151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:16.599670 18151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:16.599823 18151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:16.599913 18151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:16.634779 18151 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45881
I20260620 23:26:16.634841 18277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45881 every 8 connection(s)
I20260620 23:26:16.636145 18151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:26:16.639991 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 18151
I20260620 23:26:16.640185 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:26:16.644621 388 external_mini_cluster.cc:1428] Setting key f7afda04e76ed59c0fc0df463f41b900
I20260620 23:26:16.647481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42139
--builtin_ntp_servers=127.0.97.20:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:26:16.650731 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:16.638988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53425 (local address 127.0.97.62:42139)
0620 23:26:16.639480 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:16.639485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:16.641346 (+ 1861us) server_negotiation.cc:407] Connection header received
0620 23:26:16.643340 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:16.643347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:16.643453 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:16.643603 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:16.644867 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:16.645756 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:16.647593 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:16.648087 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:16.648238 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:16.649812 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:16.649817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:16.649821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:16.649840 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:16.649881 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:16.649894 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:16.649997 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:16.650245 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 23:26:16.650354 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":273,"threads_started":1}
I20260620 23:26:16.651896 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:16.639569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42139 (local address 127.0.97.1:53425)
0620 23:26:16.641132 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:16.641181 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:16.643009 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:16.643859 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:16.643869 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:16.643985 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:16.644632 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:16.644651 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:16.645942 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:16.645947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:16.647419 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:16.647434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:16.648369 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:16.648380 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:16.648633 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:16.649407 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:26:16.649410 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:16.649459 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:26:16.649589 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:16.649996 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:16.650011 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:16.650021 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:26:16.650244 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 23:26:16.650742 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":738,"threads_started":1}
I20260620 23:26:16.653731 18278 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42139
I20260620 23:26:16.654078 18278 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:16.654948 18278 heartbeater.cc:511] Master 127.0.97.62:42139 requested a full tablet report, sending...
I20260620 23:26:16.657351 18092 ts_manager.cc:194] Registered new tserver with Master: 9e7aec743fba4dfd833819a2f5fa06ba (127.0.97.1:45881)
I20260620 23:26:16.659600 18092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:53425
W20260620 23:26:16.998432 18282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:16.998837 18282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:16.998942 18282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:17.006739 18282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:17.006882 18282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:17.007007 18282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:26:17.016921 18282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:17.018425 18282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:17.020521 18282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:17.034804 18290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:17.034844 18288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:17.037133 18287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:17.038993 18282 server_base.cc:1034] running on GCE node
I20260620 23:26:17.039786 18282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:17.040927 18282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:17.042219 18282 hybrid_clock.cc:648] HybridClock initialized: now 1781997977042165 us; error 48 us; skew 500 ppm
I20260620 23:26:17.042624 18282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:26:17.045261 18282 webserver.cc:460] Webserver started at http://127.0.97.2:37979/ using document root <none> and password file <none>
I20260620 23:26:17.046021 18282 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:17.046144 18282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:17.046499 18282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:17.049612 18282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "64fd03997db341dab73e4856c55def6b"
format_stamp: "Formatted at 2026-06-20 23:26:17 on dist-test-slave-5kzz"
server_key: "861c883456af66d10d0ee8071d135c4f"
server_key_iv: "6431742e8d9ecac49925ec2e065a729c"
server_key_version: "encryptionkey@0"
I20260620 23:26:17.050434 18282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "64fd03997db341dab73e4856c55def6b"
format_stamp: "Formatted at 2026-06-20 23:26:17 on dist-test-slave-5kzz"
server_key: "861c883456af66d10d0ee8071d135c4f"
server_key_iv: "6431742e8d9ecac49925ec2e065a729c"
server_key_version: "encryptionkey@0"
I20260620 23:26:17.056313 18282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 23:26:17.060174 18296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:17.061736 18282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:26:17.061935 18282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "64fd03997db341dab73e4856c55def6b"
format_stamp: "Formatted at 2026-06-20 23:26:17 on dist-test-slave-5kzz"
server_key: "861c883456af66d10d0ee8071d135c4f"
server_key_iv: "6431742e8d9ecac49925ec2e065a729c"
server_key_version: "encryptionkey@0"
I20260620 23:26:17.062201 18282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:17.089362 18282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:17.090348 18282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:17.090781 18282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:17.091930 18282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:17.093699 18282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:17.093799 18282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:17.093977 18282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:17.094082 18282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:17.129504 18282 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36113
I20260620 23:26:17.129618 18408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36113 every 8 connection(s)
I20260620 23:26:17.131088 18282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:26:17.136276 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 18282
I20260620 23:26:17.136463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:26:17.136929 388 external_mini_cluster.cc:1428] Setting key ac36a21e7c854cfb2724c22d37397665
I20260620 23:26:17.139994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42139
--builtin_ntp_servers=127.0.97.20:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:26:17.146282 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.133904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50873 (local address 127.0.97.62:42139)
0620 23:26:17.134126 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:17.134133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:17.135705 (+ 1572us) server_negotiation.cc:407] Connection header received
0620 23:26:17.137791 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:17.137800 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:17.137923 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:17.138038 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:17.139560 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.140450 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.143183 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.143600 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.143716 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.145516 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:17.145521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:17.145525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:17.145546 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:17.145579 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:17.145591 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:17.145666 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:17.145941 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:26:17.146031 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:26:17.148063 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.134481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42139 (local address 127.0.97.2:50873)
0620 23:26:17.135526 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:17.135571 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:17.137454 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:17.138376 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:17.138386 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:17.138495 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:17.139362 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.139391 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.140629 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.140637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.142998 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.143020 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.143871 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.143877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.144245 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.145076 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:26:17.145082 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:17.145143 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:26:17.145274 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:17.145693 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:17.145707 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:17.145720 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:26:17.145912 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 23:26:17.146528 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":344,"threads_started":1}
I20260620 23:26:17.149788 18409 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42139
I20260620 23:26:17.150105 18409 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:17.150933 18409 heartbeater.cc:511] Master 127.0.97.62:42139 requested a full tablet report, sending...
I20260620 23:26:17.152796 18092 ts_manager.cc:194] Registered new tserver with Master: 64fd03997db341dab73e4856c55def6b (127.0.97.2:36113)
I20260620 23:26:17.153616 18092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:50873
W20260620 23:26:17.484304 18412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:17.484668 18412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:17.484822 18412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:17.492268 18412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:17.492440 18412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:17.492625 18412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:26:17.502544 18412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:17.504029 18412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:17.505959 18412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:17.520414 18417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:17.520373 18418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:17.520442 18420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:17.521225 18412 server_base.cc:1034] running on GCE node
I20260620 23:26:17.521935 18412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:17.522936 18412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:17.524186 18412 hybrid_clock.cc:648] HybridClock initialized: now 1781997977524130 us; error 35 us; skew 500 ppm
I20260620 23:26:17.524528 18412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:26:17.527046 18412 webserver.cc:460] Webserver started at http://127.0.97.3:46875/ using document root <none> and password file <none>
I20260620 23:26:17.527774 18412 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:17.527879 18412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:17.528179 18412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:17.531318 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "ebfe6c344e174cbe871828ac8070f6b7"
format_stamp: "Formatted at 2026-06-20 23:26:17 on dist-test-slave-5kzz"
server_key: "61ea6e89ea6a7f4e5710410e54af94b1"
server_key_iv: "d47ad7c25708fa7e85d4fab36c4907fd"
server_key_version: "encryptionkey@0"
I20260620 23:26:17.532121 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ebfe6c344e174cbe871828ac8070f6b7"
format_stamp: "Formatted at 2026-06-20 23:26:17 on dist-test-slave-5kzz"
server_key: "61ea6e89ea6a7f4e5710410e54af94b1"
server_key_iv: "d47ad7c25708fa7e85d4fab36c4907fd"
server_key_version: "encryptionkey@0"
I20260620 23:26:17.539354 18412 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260620 23:26:17.544938 18426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:17.546764 18412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:26:17.546989 18412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "ebfe6c344e174cbe871828ac8070f6b7"
format_stamp: "Formatted at 2026-06-20 23:26:17 on dist-test-slave-5kzz"
server_key: "61ea6e89ea6a7f4e5710410e54af94b1"
server_key_iv: "d47ad7c25708fa7e85d4fab36c4907fd"
server_key_version: "encryptionkey@0"
I20260620 23:26:17.547287 18412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:17.561772 18412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:17.562749 18412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:17.563060 18412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:17.564230 18412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:17.565953 18412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:17.566080 18412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:17.566218 18412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:17.566330 18412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:17.616940 18412 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42441
I20260620 23:26:17.616981 18538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42441 every 8 connection(s)
I20260620 23:26:17.618439 18412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:26:17.619342 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 18412
I20260620 23:26:17.619552 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:26:17.620121 388 external_mini_cluster.cc:1428] Setting key 4bc044a3c04055647d3a6b247e85be9b
I20260620 23:26:17.632297 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.621587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41311 (local address 127.0.97.62:42139)
0620 23:26:17.621771 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:17.621775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:17.623536 (+ 1761us) server_negotiation.cc:407] Connection header received
0620 23:26:17.625414 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:17.625419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:17.625517 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:17.625631 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:17.626815 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.627577 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.629203 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.629615 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.629751 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.631372 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:17.631377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:17.631381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:17.631402 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:17.631439 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:17.631453 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:17.631554 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:17.631841 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 23:26:17.631959 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:26:17.633669 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.622103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42139 (local address 127.0.97.3:41311)
0620 23:26:17.623386 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:17.623430 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:17.625129 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:17.625890 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:17.625899 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:17.625994 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:17.626603 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.626619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.627755 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.627759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.629049 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.629061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.629862 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.629869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.630116 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.631014 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:26:17.631017 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:17.631061 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:26:17.631145 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:17.631568 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:17.631580 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:17.631595 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 23:26:17.631799 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 23:26:17.632230 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":347,"threads_started":1}
I20260620 23:26:17.635953 18539 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42139
I20260620 23:26:17.636374 18539 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:17.637558 18539 heartbeater.cc:511] Master 127.0.97.62:42139 requested a full tablet report, sending...
I20260620 23:26:17.639536 18092 ts_manager.cc:194] Registered new tserver with Master: ebfe6c344e174cbe871828ac8070f6b7 (127.0.97.3:42441)
I20260620 23:26:17.640645 18092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:41311
I20260620 23:26:17.649698 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:26:17.660605 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.654417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.0.97.62:42139)
0620 23:26:17.654780 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:17.654788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:17.655090 (+ 302us) server_negotiation.cc:407] Connection header received
0620 23:26:17.655314 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:17.655318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:17.655385 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:17.655500 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:17.656004 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.656746 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.658132 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.658584 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.658760 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.659868 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:17.659873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:17.659876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:17.659895 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:17.659927 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:17.659940 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:17.660039 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:17.660180 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 23:26:17.660278 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260620 23:26:17.663264 18278 heartbeater.cc:503] Master 127.0.97.62:42139 was elected leader, sending a full tablet report...
I20260620 23:26:17.665437 18091 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:26:17.672917 18091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:26:17.710367 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.688109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36113 (local address 127.0.0.1:43240)
0620 23:26:17.700558 (+ 12449us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:17.700582 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:17.700724 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:17.701994 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:17.702005 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:17.702039 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:17.702666 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.702711 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.704827 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.704831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.707078 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.707093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.709369 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.709374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.709611 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.709667 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 23:26:17.709792 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 23:26:17.710014 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12267,"thread_start_us":11941,"threads_started":1}
I20260620 23:26:17.711074 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.688171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.0.97.2:36113)
0620 23:26:17.701232 (+ 13061us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:17.701242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:17.701278 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:26:17.701390 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:17.701400 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:17.701695 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:17.702958 (+ 1263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:17.703219 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.704632 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.707292 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.708808 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.708948 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.709244 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:17.710507 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0620 23:26:17.710729 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12911,"thread_start_us":12810,"threads_started":1}
I20260620 23:26:17.711202 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.688094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42441 (local address 127.0.0.1:35948)
0620 23:26:17.699663 (+ 11569us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:17.699707 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:17.699884 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:17.703102 (+ 3218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:17.703118 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:17.703151 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:17.703505 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.703518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.705571 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.705577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.707714 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.707725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.709810 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.709816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.709977 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.710035 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 23:26:17.710124 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:26:17.710264 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11260,"mutex_wait_us":115,"thread_start_us":10056,"threads_started":1}
I20260620 23:26:17.711339 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.688858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45881 (local address 127.0.0.1:58204)
0620 23:26:17.702625 (+ 13767us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:17.702647 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:17.702818 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:17.703941 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:17.703945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:17.703971 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:17.704412 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.704425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.706111 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.706118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.708203 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.708220 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.710506 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.710512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.710721 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.710752 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:26:17.710839 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:26:17.710965 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13630,"mutex_wait_us":4,"thread_start_us":13494,"threads_started":1}
I20260620 23:26:17.711575 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.688772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35948 (local address 127.0.97.3:42441)
0620 23:26:17.702304 (+ 13532us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:17.702316 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:17.702353 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:26:17.702454 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:17.702464 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:17.702793 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:17.702986 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:17.703709 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.705413 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.707912 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.709248 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.709387 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.709664 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:17.710213 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 23:26:17.710412 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13241,"thread_start_us":13097,"threads_started":1}
I20260620 23:26:17.712026 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.689485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.0.97.1:45881)
0620 23:26:17.701678 (+ 12193us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:17.701686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:17.702953 (+ 1267us) server_negotiation.cc:407] Connection header received
0620 23:26:17.703183 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:17.703193 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:17.703446 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:17.703656 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:17.704627 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.705956 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.708420 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.709952 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.710097 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.710373 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:17.711452 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0620 23:26:17.711660 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12012,"thread_start_us":11895,"threads_started":1}
I20260620 23:26:17.715260 18344 tablet_service.cc:1467] Processing CreateTablet for tablet a3d27fa207c248668bc8c98912d7cc95 (DEFAULT_TABLE table=test-table [id=e51d41083ac34704adf5e3395e72d25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:17.715790 18474 tablet_service.cc:1467] Processing CreateTablet for tablet a3d27fa207c248668bc8c98912d7cc95 (DEFAULT_TABLE table=test-table [id=e51d41083ac34704adf5e3395e72d25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:17.715788 18213 tablet_service.cc:1467] Processing CreateTablet for tablet a3d27fa207c248668bc8c98912d7cc95 (DEFAULT_TABLE table=test-table [id=e51d41083ac34704adf5e3395e72d25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:17.716813 18344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d27fa207c248668bc8c98912d7cc95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:17.717204 18213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d27fa207c248668bc8c98912d7cc95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:17.717226 18474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d27fa207c248668bc8c98912d7cc95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:17.728832 18557 tablet_bootstrap.cc:492] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b: Bootstrap starting.
I20260620 23:26:17.732568 18558 tablet_bootstrap.cc:492] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba: Bootstrap starting.
I20260620 23:26:17.734311 18557 tablet_bootstrap.cc:654] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:17.734660 18559 tablet_bootstrap.cc:492] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7: Bootstrap starting.
I20260620 23:26:17.736428 18557 log.cc:824] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:17.736706 18558 tablet_bootstrap.cc:654] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:17.737864 18558 log.cc:824] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:17.738269 18559 tablet_bootstrap.cc:654] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:17.739580 18559 log.cc:824] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:17.740002 18557 tablet_bootstrap.cc:492] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b: No bootstrap required, opened a new log
I20260620 23:26:17.740317 18557 ts_tablet_manager.cc:1397] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.006s
I20260620 23:26:17.741400 18558 tablet_bootstrap.cc:492] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba: No bootstrap required, opened a new log
I20260620 23:26:17.741695 18558 ts_tablet_manager.cc:1397] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 23:26:17.742615 18559 tablet_bootstrap.cc:492] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7: No bootstrap required, opened a new log
I20260620 23:26:17.742925 18559 ts_tablet_manager.cc:1397] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 23:26:17.750988 18559 raft_consensus.cc:348] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd03997db341dab73e4856c55def6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36113 } } peers { permanent_uuid: "ebfe6c344e174cbe871828ac8070f6b7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42441 } } peers { permanent_uuid: "9e7aec743fba4dfd833819a2f5fa06ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45881 } }
I20260620 23:26:17.751595 18559 raft_consensus.cc:374] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:17.751708 18559 raft_consensus.cc:729] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfe6c344e174cbe871828ac8070f6b7, State: Initialized, Role: FOLLOWER
I20260620 23:26:17.751734 18557 raft_consensus.cc:348] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd03997db341dab73e4856c55def6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36113 } } peers { permanent_uuid: "ebfe6c344e174cbe871828ac8070f6b7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42441 } } peers { permanent_uuid: "9e7aec743fba4dfd833819a2f5fa06ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45881 } }
I20260620 23:26:17.752179 18557 raft_consensus.cc:374] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:17.752276 18557 raft_consensus.cc:729] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd03997db341dab73e4856c55def6b, State: Initialized, Role: FOLLOWER
I20260620 23:26:17.752409 18559 consensus_queue.cc:260] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd03997db341dab73e4856c55def6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36113 } } peers { permanent_uuid: "ebfe6c344e174cbe871828ac8070f6b7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42441 } } peers { permanent_uuid: "9e7aec743fba4dfd833819a2f5fa06ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45881 } }
I20260620 23:26:17.752805 18558 raft_consensus.cc:348] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd03997db341dab73e4856c55def6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36113 } } peers { permanent_uuid: "ebfe6c344e174cbe871828ac8070f6b7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42441 } } peers { permanent_uuid: "9e7aec743fba4dfd833819a2f5fa06ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45881 } }
I20260620 23:26:17.752970 18557 consensus_queue.cc:260] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd03997db341dab73e4856c55def6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36113 } } peers { permanent_uuid: "ebfe6c344e174cbe871828ac8070f6b7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42441 } } peers { permanent_uuid: "9e7aec743fba4dfd833819a2f5fa06ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45881 } }
I20260620 23:26:17.753259 18558 raft_consensus.cc:374] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:17.753364 18558 raft_consensus.cc:729] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7aec743fba4dfd833819a2f5fa06ba, State: Initialized, Role: FOLLOWER
I20260620 23:26:17.753925 18539 heartbeater.cc:503] Master 127.0.97.62:42139 was elected leader, sending a full tablet report...
I20260620 23:26:17.754185 18558 consensus_queue.cc:260] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd03997db341dab73e4856c55def6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36113 } } peers { permanent_uuid: "ebfe6c344e174cbe871828ac8070f6b7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42441 } } peers { permanent_uuid: "9e7aec743fba4dfd833819a2f5fa06ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45881 } }
I20260620 23:26:17.754442 18559 ts_tablet_manager.cc:1428] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260620 23:26:17.758245 18558 ts_tablet_manager.cc:1428] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260620 23:26:17.760543 18409 heartbeater.cc:503] Master 127.0.97.62:42139 was elected leader, sending a full tablet report...
I20260620 23:26:17.762017 18557 ts_tablet_manager.cc:1428] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b: Time spent starting tablet: real 0.022s user 0.010s sys 0.012s
I20260620 23:26:17.847362 18563 raft_consensus.cc:482] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:26:17.847726 18563 raft_consensus.cc:504] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd03997db341dab73e4856c55def6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36113 } } peers { permanent_uuid: "ebfe6c344e174cbe871828ac8070f6b7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42441 } } peers { permanent_uuid: "9e7aec743fba4dfd833819a2f5fa06ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45881 } }
I20260620 23:26:17.849803 18563 leader_election.cc:288] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64fd03997db341dab73e4856c55def6b (127.0.97.2:36113), 9e7aec743fba4dfd833819a2f5fa06ba (127.0.97.1:45881)
I20260620 23:26:17.856395 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.850661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33867 (local address 127.0.97.2:36113)
0620 23:26:17.850912 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:17.850921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:17.850942 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:26:17.851143 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:17.851152 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:17.851277 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:17.851394 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:17.851808 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.852727 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.854619 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.855462 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.855616 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.855712 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:17.855989 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 23:26:17.856119 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:26:17.856426 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.850186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36113 (local address 127.0.97.3:33867)
0620 23:26:17.850558 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:17.850578 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:17.850746 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:17.851442 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:17.851447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:17.851473 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:17.851641 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.851649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.852849 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.852855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.854312 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.854325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.855630 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.855636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.855811 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.855836 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 23:26:17.855953 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:26:17.856101 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260620 23:26:17.856442 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.850374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47763 (local address 127.0.97.1:45881)
0620 23:26:17.850611 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:17.850620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:17.851237 (+ 617us) server_negotiation.cc:407] Connection header received
0620 23:26:17.851465 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:17.851474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:17.851594 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:17.851703 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:17.852245 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.852997 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.854794 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.855576 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.855725 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.855822 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:17.856077 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 23:26:17.856206 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 23:26:17.856799 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.850226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45881 (local address 127.0.97.3:47763)
0620 23:26:17.851129 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:17.851143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:17.851290 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:17.851717 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:17.851721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:17.851744 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:17.852071 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.852084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.853164 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.853169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.854595 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:17.854606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.855748 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.855754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:17.855919 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.855942 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:26:17.856049 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 23:26:17.856187 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":1,"thread_start_us":601,"threads_started":1}
I20260620 23:26:17.857584 18364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d27fa207c248668bc8c98912d7cc95" candidate_uuid: "ebfe6c344e174cbe871828ac8070f6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd03997db341dab73e4856c55def6b" is_pre_election: true
I20260620 23:26:17.857962 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d27fa207c248668bc8c98912d7cc95" candidate_uuid: "ebfe6c344e174cbe871828ac8070f6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e7aec743fba4dfd833819a2f5fa06ba" is_pre_election: true
I20260620 23:26:17.858124 18364 raft_consensus.cc:2454] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebfe6c344e174cbe871828ac8070f6b7 in term 0.
I20260620 23:26:17.858469 18233 raft_consensus.cc:2454] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebfe6c344e174cbe871828ac8070f6b7 in term 0.
I20260620 23:26:17.858896 18429 leader_election.cc:302] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64fd03997db341dab73e4856c55def6b, ebfe6c344e174cbe871828ac8070f6b7; no voters:
I20260620 23:26:17.859325 18563 raft_consensus.cc:2781] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:26:17.859480 18563 raft_consensus.cc:482] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:26:17.859596 18563 raft_consensus.cc:3037] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:17.861829 18563 raft_consensus.cc:504] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd03997db341dab73e4856c55def6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36113 } } peers { permanent_uuid: "ebfe6c344e174cbe871828ac8070f6b7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42441 } } peers { permanent_uuid: "9e7aec743fba4dfd833819a2f5fa06ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45881 } }
I20260620 23:26:17.862455 18563 leader_election.cc:288] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [CANDIDATE]: Term 1 election: Requested vote from peers 64fd03997db341dab73e4856c55def6b (127.0.97.2:36113), 9e7aec743fba4dfd833819a2f5fa06ba (127.0.97.1:45881)
I20260620 23:26:17.862993 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d27fa207c248668bc8c98912d7cc95" candidate_uuid: "ebfe6c344e174cbe871828ac8070f6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e7aec743fba4dfd833819a2f5fa06ba"
I20260620 23:26:17.863009 18364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d27fa207c248668bc8c98912d7cc95" candidate_uuid: "ebfe6c344e174cbe871828ac8070f6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd03997db341dab73e4856c55def6b"
I20260620 23:26:17.863188 18233 raft_consensus.cc:3037] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:17.863231 18364 raft_consensus.cc:3037] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:17.865510 18233 raft_consensus.cc:2454] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebfe6c344e174cbe871828ac8070f6b7 in term 1.
I20260620 23:26:17.865645 18364 raft_consensus.cc:2454] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebfe6c344e174cbe871828ac8070f6b7 in term 1.
I20260620 23:26:17.866051 18427 leader_election.cc:302] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e7aec743fba4dfd833819a2f5fa06ba, ebfe6c344e174cbe871828ac8070f6b7; no voters:
I20260620 23:26:17.866371 18563 raft_consensus.cc:2781] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:17.866935 18563 raft_consensus.cc:686] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 1 LEADER]: Becoming Leader. State: Replica: ebfe6c344e174cbe871828ac8070f6b7, State: Running, Role: LEADER
I20260620 23:26:17.867444 18563 consensus_queue.cc:237] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd03997db341dab73e4856c55def6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36113 } } peers { permanent_uuid: "ebfe6c344e174cbe871828ac8070f6b7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42441 } } peers { permanent_uuid: "9e7aec743fba4dfd833819a2f5fa06ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45881 } }
W20260620 23:26:17.870568 18540 tablet.cc:2369] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:26:17.872978 18092 catalog_manager.cc:5495] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebfe6c344e174cbe871828ac8070f6b7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ebfe6c344e174cbe871828ac8070f6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd03997db341dab73e4856c55def6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebfe6c344e174cbe871828ac8070f6b7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e7aec743fba4dfd833819a2f5fa06ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45881 } health_report { overall_health: UNKNOWN } } }
W20260620 23:26:17.883431 18410 tablet.cc:2369] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:26:17.889015 18279 tablet.cc:2369] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:26:17.900600 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.895065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35950 (local address 127.0.97.3:42441)
0620 23:26:17.895243 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:17.895247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:17.895265 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:26:17.895356 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:17.895360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:17.895455 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:17.895615 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:26:17.896123 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.896882 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.898353 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.898876 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.898997 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.899506 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:26:17.900058 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:26:17.900172 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:17.900326 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:26:17.900394 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:26:17.910580 18233 raft_consensus.cc:1261] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba [term 1 FOLLOWER]: Refusing update from remote peer ebfe6c344e174cbe871828ac8070f6b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:26:17.910753 18364 raft_consensus.cc:1261] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b [term 1 FOLLOWER]: Refusing update from remote peer ebfe6c344e174cbe871828ac8070f6b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:26:17.911732 18567 consensus_queue.cc:1035] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e7aec743fba4dfd833819a2f5fa06ba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:26:17.912171 18563 consensus_queue.cc:1035] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd03997db341dab73e4856c55def6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:26:17.923002 18570 mvcc.cc:205] Tried to move back new op lower bound from 7299063717511348224 to 7299063717349351424. Current Snapshot: MvccSnapshot[applied={T|T < 7299063717511348224}]
I20260620 23:26:17.929603 18571 mvcc.cc:205] Tried to move back new op lower bound from 7299063717511348224 to 7299063717349351424. Current Snapshot: MvccSnapshot[applied={T|T < 7299063717511348224}]
I20260620 23:26:17.946192 18092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:26:17.946508 18092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:26:17.950392 18092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 495872a5c0a0478baaf4dd96e488016d: Sending DeleteTablet for 3 replicas of tablet a3d27fa207c248668bc8c98912d7cc95
I20260620 23:26:17.951718 18474 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d27fa207c248668bc8c98912d7cc95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:17 UTC) from {username='slave'} at 127.0.0.1:35948
I20260620 23:26:17.951731 18344 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d27fa207c248668bc8c98912d7cc95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:17 UTC) from {username='slave'} at 127.0.0.1:43240
I20260620 23:26:17.953234 18213 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d27fa207c248668bc8c98912d7cc95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:17 UTC) from {username='slave'} at 127.0.0.1:58204
I20260620 23:26:17.955013 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 18151
I20260620 23:26:17.967305 18581 tablet_replica.cc:330] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b: stopping tablet replica
I20260620 23:26:17.967859 18581 raft_consensus.cc:2229] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:26:17.968436 18581 raft_consensus.cc:2258] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:17.969307 18580 tablet_replica.cc:330] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7: stopping tablet replica
I20260620 23:26:17.969756 18582 tablet_replica.cc:330] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba: stopping tablet replica
I20260620 23:26:17.969760 18580 raft_consensus.cc:2229] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:26:17.970139 18582 raft_consensus.cc:2229] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:26:17.970436 18580 raft_consensus.cc:2258] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:17.971688 18582 raft_consensus.cc:2258] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:17.974300 18580 ts_tablet_manager.cc:1905] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:17.974291 18582 ts_tablet_manager.cc:1905] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:17.974591 18581 ts_tablet_manager.cc:1905] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:17.981801 18581 ts_tablet_manager.cc:1918] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:17.981822 18580 ts_tablet_manager.cc:1918] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:17.981806 18582 ts_tablet_manager.cc:1918] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:17.981995 18581 log.cc:1196] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/a3d27fa207c248668bc8c98912d7cc95
I20260620 23:26:17.982000 18580 log.cc:1196] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/a3d27fa207c248668bc8c98912d7cc95
I20260620 23:26:17.982234 18582 log.cc:1196] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/a3d27fa207c248668bc8c98912d7cc95
I20260620 23:26:17.982590 18580 ts_tablet_manager.cc:1939] T a3d27fa207c248668bc8c98912d7cc95 P ebfe6c344e174cbe871828ac8070f6b7: Deleting consensus metadata
I20260620 23:26:17.982630 18581 ts_tablet_manager.cc:1939] T a3d27fa207c248668bc8c98912d7cc95 P 64fd03997db341dab73e4856c55def6b: Deleting consensus metadata
I20260620 23:26:17.983024 18582 ts_tablet_manager.cc:1939] T a3d27fa207c248668bc8c98912d7cc95 P 9e7aec743fba4dfd833819a2f5fa06ba: Deleting consensus metadata
I20260620 23:26:17.985176 18076 catalog_manager.cc:4841] TS 9e7aec743fba4dfd833819a2f5fa06ba (127.0.97.1:45881): tablet a3d27fa207c248668bc8c98912d7cc95 (table test-table [id=e51d41083ac34704adf5e3395e72d25d]) successfully deleted
I20260620 23:26:17.985569 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:17.956408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.97.1:45881)
0620 23:26:17.956638 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:17.956647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:17.960683 (+ 4036us) server_negotiation.cc:407] Connection header received
0620 23:26:17.961644 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:17.961650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:17.961792 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:17.961882 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:17.963010 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.964239 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.971186 (+ 6947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:17.971946 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:17.972095 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:17.983309 (+ 11214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:17.983324 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:17.983344 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:17.984189 (+ 845us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:17.984257 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:17.984286 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:17.984416 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:17.985076 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 23:26:17.985201 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:26:17.985888 18077 catalog_manager.cc:4841] TS ebfe6c344e174cbe871828ac8070f6b7 (127.0.97.3:42441): tablet a3d27fa207c248668bc8c98912d7cc95 (table test-table [id=e51d41083ac34704adf5e3395e72d25d]) successfully deleted
I20260620 23:26:17.986369 18273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:17.987402 18078 catalog_manager.cc:4841] TS 64fd03997db341dab73e4856c55def6b (127.0.97.2:36113): tablet a3d27fa207c248668bc8c98912d7cc95 (table test-table [id=e51d41083ac34704adf5e3395e72d25d]) successfully deleted
I20260620 23:26:18.113526 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 18151
I20260620 23:26:18.133469 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 18282
I20260620 23:26:18.142608 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:18.134417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.0.97.2:36113)
0620 23:26:18.134665 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:18.134763 (+ 98us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:18.134786 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:26:18.134848 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:18.134854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:18.134975 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:18.135089 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:18.135550 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:18.136804 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:18.138732 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:18.139426 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:18.139567 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:18.140853 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:18.140865 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:18.140885 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:18.141713 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:18.141786 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:18.141813 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:18.141932 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:18.142093 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 23:26:18.142214 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260620 23:26:18.143421 18404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:18.263916 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 18282
I20260620 23:26:18.282256 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 18412
I20260620 23:26:18.290639 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:18.283115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.0.97.3:42441)
0620 23:26:18.283338 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:18.283343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:18.283383 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:26:18.283660 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:18.283664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:18.283816 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:18.283939 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:18.284490 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:18.285295 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:18.287242 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:18.287696 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:18.287801 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:18.289062 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:18.289070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:18.289085 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:18.289751 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:18.289800 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:18.289820 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:18.289907 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:18.290105 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 23:26:18.290188 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 23:26:18.291463 18534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:18.414265 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 18412
I20260620 23:26:18.433974 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 18061
I20260620 23:26:18.435189 18122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:18.547256 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 18061
2026-06-20T23:26:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3048 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:18 dist-test-slave-5kzz krb5kdc[18590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:18 dist-test-slave-5kzz krb5kdc[18590](info): set up 2 sockets
Jun 20 23:26:18 dist-test-slave-5kzz krb5kdc[18590](info): commencing operation
krb5kdc: starting...
W20260620 23:26:19.659085 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:19 dist-test-slave-5kzz krb5kdc[18590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:26:19.910930 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35805
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:20.250514 18603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:20.250907 18603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:20.251071 18603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:20.258556 18603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:20.258744 18603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:20.258837 18603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:20.258927 18603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:20.258983 18603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:20.269253 18603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:20.270726 18603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:20.272156 18603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:20.284691 18611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:20.284623 18609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:20.285157 18608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:20.285517 18603 server_base.cc:1034] running on GCE node
I20260620 23:26:20.286432 18603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:20.288110 18603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:20.289351 18603 hybrid_clock.cc:648] HybridClock initialized: now 1781997980289279 us; error 71 us; skew 500 ppm
I20260620 23:26:20.289791 18603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:20 dist-test-slave-5kzz krb5kdc[18590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:20.293217 18603 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:26:20.294947 18603 webserver.cc:460] Webserver started at http://127.0.97.62:38935/ using document root <none> and password file <none>
I20260620 23:26:20.295727 18603 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:20.295879 18603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:20.296217 18603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:20.299470 18603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "6d545282b5494399b888d92e2a25d78a"
format_stamp: "Formatted at 2026-06-20 23:26:20 on dist-test-slave-5kzz"
server_key: "93dd3396bc230d316313ad701eeefafb"
server_key_iv: "cbab15a918fcfb4dfacdc40e0fb141de"
server_key_version: "encryptionkey@0"
I20260620 23:26:20.300302 18603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "6d545282b5494399b888d92e2a25d78a"
format_stamp: "Formatted at 2026-06-20 23:26:20 on dist-test-slave-5kzz"
server_key: "93dd3396bc230d316313ad701eeefafb"
server_key_iv: "cbab15a918fcfb4dfacdc40e0fb141de"
server_key_version: "encryptionkey@0"
I20260620 23:26:20.305900 18603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:26:20.309614 18618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:20.311304 18603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:26:20.311549 18603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "6d545282b5494399b888d92e2a25d78a"
format_stamp: "Formatted at 2026-06-20 23:26:20 on dist-test-slave-5kzz"
server_key: "93dd3396bc230d316313ad701eeefafb"
server_key_iv: "cbab15a918fcfb4dfacdc40e0fb141de"
server_key_version: "encryptionkey@0"
I20260620 23:26:20.311779 18603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:20.325840 18603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:20.330229 18603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:20.330575 18603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:20.350003 18603 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35805
I20260620 23:26:20.350045 18669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35805 every 8 connection(s)
I20260620 23:26:20.351557 18603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:20.355948 18670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:20.356634 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 18603
I20260620 23:26:20.356842 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:20.357280 388 external_mini_cluster.cc:1428] Setting key b9f719bc9609271b4939875a34c4d0d1
I20260620 23:26:20.372402 18670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a: Bootstrap starting.
Jun 20 23:26:20 dist-test-slave-5kzz krb5kdc[18590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:20.376324 18670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:20.377884 18670 log.cc:824] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:20.381795 18670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a: No bootstrap required, opened a new log
I20260620 23:26:20.385483 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:20.359332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.0.97.62:35805)
0620 23:26:20.360219 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:20.360237 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:20.360302 (+ 65us) server_negotiation.cc:407] Connection header received
0620 23:26:20.363523 (+ 3221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:20.363558 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:20.364212 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:20.364743 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:20.366106 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:20.368354 (+ 2248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:20.369782 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:20.371027 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:20.371237 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:20.375968 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:20.375981 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:20.376003 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:20.376064 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:20.380441 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:20.381175 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:20.381184 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:20.381194 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:20.381439 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:20.381795 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:20.381800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:20.381804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:20.382637 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:20.383052 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:20.383391 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 23:26:20.383883 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":370,"threads_started":1}
I20260620 23:26:20.392042 18670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d545282b5494399b888d92e2a25d78a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } }
I20260620 23:26:20.392441 18670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:20.392522 18670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d545282b5494399b888d92e2a25d78a, State: Initialized, Role: FOLLOWER
I20260620 23:26:20.393067 18670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d545282b5494399b888d92e2a25d78a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } }
I20260620 23:26:20.393289 18670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:20.393389 18670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:20.393523 18670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:20.396320 18670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d545282b5494399b888d92e2a25d78a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } }
I20260620 23:26:20.396827 18670 leader_election.cc:302] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d545282b5494399b888d92e2a25d78a; no voters:
I20260620 23:26:20.397363 18670 leader_election.cc:288] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:20.397526 18675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:20.398332 18675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [term 1 LEADER]: Becoming Leader. State: Replica: 6d545282b5494399b888d92e2a25d78a, State: Running, Role: LEADER
I20260620 23:26:20.398893 18675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d545282b5494399b888d92e2a25d78a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } }
I20260620 23:26:20.399765 18670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:20.401940 18676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d545282b5494399b888d92e2a25d78a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d545282b5494399b888d92e2a25d78a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } } }
I20260620 23:26:20.402438 18676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:20.406284 18677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d545282b5494399b888d92e2a25d78a. Latest consensus state: current_term: 1 leader_uuid: "6d545282b5494399b888d92e2a25d78a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d545282b5494399b888d92e2a25d78a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } } }
I20260620 23:26:20.406725 18677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:20.408890 18683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:20.415400 18683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:20.425313 18683 catalog_manager.cc:1347] Generated new cluster ID: a3a8496a6c764ad7b5d28d2525e8b54b
I20260620 23:26:20.425454 18683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:20.455996 18683 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:20.457544 18683 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:20.471964 18683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a: Generated new TSK 0
I20260620 23:26:20.472947 18683 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:26:20.554828 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:20.906148 18696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:20.906445 18696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:20.906615 18696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:20.914987 18696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:20.915154 18696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:20.915333 18696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:26:20.926465 18696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:20.927938 18696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:20.929319 18696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:20.943878 18704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:20.943920 18701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:20.943883 18702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:20.944454 18696 server_base.cc:1034] running on GCE node
I20260620 23:26:20.945312 18696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:20.946296 18696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:20.947506 18696 hybrid_clock.cc:648] HybridClock initialized: now 1781997980947426 us; error 70 us; skew 500 ppm
I20260620 23:26:20.947808 18696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:20 dist-test-slave-5kzz krb5kdc[18590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:20.951265 18696 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:26:20.953032 18696 webserver.cc:460] Webserver started at http://127.0.97.1:40035/ using document root <none> and password file <none>
I20260620 23:26:20.953770 18696 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:20.953923 18696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:20.954252 18696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:20.957417 18696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "4b9bff7f8b014b14a551605474f2bc25"
format_stamp: "Formatted at 2026-06-20 23:26:20 on dist-test-slave-5kzz"
server_key: "cccf6d7aad4c536f83701cecf3adbcb6"
server_key_iv: "acfeeb151f93e74d4fdc970d17d0cfba"
server_key_version: "encryptionkey@0"
I20260620 23:26:20.958266 18696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4b9bff7f8b014b14a551605474f2bc25"
format_stamp: "Formatted at 2026-06-20 23:26:20 on dist-test-slave-5kzz"
server_key: "cccf6d7aad4c536f83701cecf3adbcb6"
server_key_iv: "acfeeb151f93e74d4fdc970d17d0cfba"
server_key_version: "encryptionkey@0"
I20260620 23:26:20.964147 18696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:26:20.968106 18711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:20.969599 18696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 23:26:20.969905 18696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "4b9bff7f8b014b14a551605474f2bc25"
format_stamp: "Formatted at 2026-06-20 23:26:20 on dist-test-slave-5kzz"
server_key: "cccf6d7aad4c536f83701cecf3adbcb6"
server_key_iv: "acfeeb151f93e74d4fdc970d17d0cfba"
server_key_version: "encryptionkey@0"
I20260620 23:26:20.970141 18696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:20.999707 18696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:21.004541 18696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:21.004873 18696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:21.005911 18696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:21.007483 18696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:21.007586 18696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:21.007745 18696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:21.007818 18696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:21.042575 18696 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36047
I20260620 23:26:21.042625 18823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36047 every 8 connection(s)
I20260620 23:26:21.043942 18696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:26:21.051822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 18696
I20260620 23:26:21.052057 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:26:21.052497 388 external_mini_cluster.cc:1428] Setting key e6e5475087667945a95a36c6d987969c
Jun 20 23:26:21 dist-test-slave-5kzz krb5kdc[18590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:21.066293 18826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:21.046878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54189 (local address 127.0.97.62:35805)
0620 23:26:21.047423 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:21.047428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:21.048273 (+ 845us) server_negotiation.cc:407] Connection header received
0620 23:26:21.050163 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:21.050170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:21.050272 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:21.050410 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:21.052107 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:21.052941 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:21.054788 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:21.055187 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:21.055288 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:21.059795 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:21.059800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:21.059803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:21.059840 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:21.062656 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:21.063533 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:21.063539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:21.063544 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:21.063706 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:21.064193 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:21.064201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:21.064207 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:21.064616 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:21.064775 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:21.065546 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 23:26:21.065813 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":300,"threads_started":1}
I20260620 23:26:21.067024 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:21.047313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35805 (local address 127.0.97.1:54189)
0620 23:26:21.048090 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:21.048129 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:21.049845 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:21.050649 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:21.050658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:21.051263 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:21.051870 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:21.051897 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:21.053144 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:21.053150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:21.054623 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:21.054640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:21.055431 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:21.055440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:21.055799 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:21.056614 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:26:21.056650 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:21.059541 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:21.062854 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:21.062863 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:21.062879 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:21.063364 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:21.063846 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:21.063850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:21.063852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:21.064063 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:21.064749 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:21.064755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:21.065126 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 23:26:21.065370 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 23:26:21.065789 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":334,"threads_started":1}
I20260620 23:26:21.068840 18824 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35805
I20260620 23:26:21.069198 18824 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:21.069990 18824 heartbeater.cc:511] Master 127.0.97.62:35805 requested a full tablet report, sending...
I20260620 23:26:21.072290 18635 ts_manager.cc:194] Registered new tserver with Master: 4b9bff7f8b014b14a551605474f2bc25 (127.0.97.1:36047)
I20260620 23:26:21.073871 18635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54189
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:26:21.110224 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:21.454156 18830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:21.454516 18830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:21.454712 18830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:21.462026 18830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:21.462212 18830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:21.462410 18830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:26:21.472518 18830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:21.473903 18830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:21.475651 18830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:21.489019 18835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:21.489027 18838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:21.489061 18836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:21.489996 18830 server_base.cc:1034] running on GCE node
I20260620 23:26:21.490649 18830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:21.491775 18830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:21.493008 18830 hybrid_clock.cc:648] HybridClock initialized: now 1781997981492947 us; error 47 us; skew 500 ppm
I20260620 23:26:21.493324 18830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:21 dist-test-slave-5kzz krb5kdc[18590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:21.496655 18830 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:26:21.498126 18830 webserver.cc:460] Webserver started at http://127.0.97.2:39903/ using document root <none> and password file <none>
I20260620 23:26:21.498895 18830 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:21.499038 18830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:21.499349 18830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:21.502799 18830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "b6c4342160534b75bbf4c86349b81673"
format_stamp: "Formatted at 2026-06-20 23:26:21 on dist-test-slave-5kzz"
server_key: "5cd3fe89e6d969231faf5411bb400fb8"
server_key_iv: "03797e684aa32f2a8e5d09abfee36be1"
server_key_version: "encryptionkey@0"
I20260620 23:26:21.503675 18830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b6c4342160534b75bbf4c86349b81673"
format_stamp: "Formatted at 2026-06-20 23:26:21 on dist-test-slave-5kzz"
server_key: "5cd3fe89e6d969231faf5411bb400fb8"
server_key_iv: "03797e684aa32f2a8e5d09abfee36be1"
server_key_version: "encryptionkey@0"
I20260620 23:26:21.509169 18830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:26:21.512912 18845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:21.514484 18830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:26:21.514668 18830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "b6c4342160534b75bbf4c86349b81673"
format_stamp: "Formatted at 2026-06-20 23:26:21 on dist-test-slave-5kzz"
server_key: "5cd3fe89e6d969231faf5411bb400fb8"
server_key_iv: "03797e684aa32f2a8e5d09abfee36be1"
server_key_version: "encryptionkey@0"
I20260620 23:26:21.514978 18830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:21.584227 18830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:21.588856 18830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:21.589166 18830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:21.590139 18830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:21.591646 18830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:21.591744 18830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:21.591895 18830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:21.591964 18830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:21.630826 18830 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33827
I20260620 23:26:21.630976 18957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33827 every 8 connection(s)
I20260620 23:26:21.632156 18830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:26:21.636548 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 18830
I20260620 23:26:21.636736 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:26:21.637179 388 external_mini_cluster.cc:1428] Setting key 76f9d4a3ccf3430935857e3b916a2592
Jun 20 23:26:21 dist-test-slave-5kzz krb5kdc[18590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:21.653043 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:21.634843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55453 (local address 127.0.97.62:35805)
0620 23:26:21.635291 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:21.635295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:21.636132 (+ 837us) server_negotiation.cc:407] Connection header received
0620 23:26:21.638237 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:21.638245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:21.638344 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:21.638466 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:21.640197 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:21.640940 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:21.642526 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:21.643020 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:21.643167 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:21.646732 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:21.646745 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:21.646749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:21.646791 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:21.649639 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:21.650540 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:21.650553 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:21.650556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:21.650734 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:21.651182 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:21.651188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:21.651192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:21.651498 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:21.651649 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:21.652405 (+ 756us) server_negotiation.cc:299] Negotiation successful
0620 23:26:21.652643 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":232,"threads_started":1}
I20260620 23:26:21.654060 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:21.635292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35805 (local address 127.0.97.2:55453)
0620 23:26:21.635968 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:21.636010 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:21.637969 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:21.638767 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:21.638776 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:21.639449 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:21.640018 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:21.640039 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:21.641092 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:21.641097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:21.642356 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:21.642368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:21.643263 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:21.643270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:21.643509 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:21.644227 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:26:21.644249 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:21.646438 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:21.649858 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:21.649868 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:21.649892 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:21.650391 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:21.650856 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:21.650861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:21.650865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:21.651056 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:21.651637 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:21.651645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:21.652024 (+ 379us) client_negotiation.cc:769] Sending connection context
0620 23:26:21.652314 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 23:26:21.652885 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":271,"threads_started":1}
I20260620 23:26:21.655654 18958 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35805
I20260620 23:26:21.655997 18958 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:21.656759 18958 heartbeater.cc:511] Master 127.0.97.62:35805 requested a full tablet report, sending...
I20260620 23:26:21.658459 18635 ts_manager.cc:194] Registered new tserver with Master: b6c4342160534b75bbf4c86349b81673 (127.0.97.2:33827)
I20260620 23:26:21.659760 18635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55453
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:26:21.694202 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:22.038630 18964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:22.039009 18964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:22.039175 18964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:22.046999 18964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:22.047101 18964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:22.047289 18964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:26:22.057238 18964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:22.058511 18964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:22.059826 18964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:22.073628 18970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:22.073671 18972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:22.073915 18969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:22.074748 18964 server_base.cc:1034] running on GCE node
I20260620 23:26:22.075416 18964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:22.076370 18964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:22.077010 18824 heartbeater.cc:503] Master 127.0.97.62:35805 was elected leader, sending a full tablet report...
I20260620 23:26:22.077589 18964 hybrid_clock.cc:648] HybridClock initialized: now 1781997982077543 us; error 29 us; skew 500 ppm
I20260620 23:26:22.077978 18964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:22 dist-test-slave-5kzz krb5kdc[18590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:22.081454 18964 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:26:22.083099 18964 webserver.cc:460] Webserver started at http://127.0.97.3:44925/ using document root <none> and password file <none>
I20260620 23:26:22.083768 18964 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:22.083875 18964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:22.084227 18964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:22.087409 18964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "a9e998e5513f42a681a7af15fd4b254d"
format_stamp: "Formatted at 2026-06-20 23:26:22 on dist-test-slave-5kzz"
server_key: "a833f90fd18d3c7a59f998bb708bfad1"
server_key_iv: "445a000cf37e70ddbf10e48ba5a00b64"
server_key_version: "encryptionkey@0"
I20260620 23:26:22.088182 18964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a9e998e5513f42a681a7af15fd4b254d"
format_stamp: "Formatted at 2026-06-20 23:26:22 on dist-test-slave-5kzz"
server_key: "a833f90fd18d3c7a59f998bb708bfad1"
server_key_iv: "445a000cf37e70ddbf10e48ba5a00b64"
server_key_version: "encryptionkey@0"
I20260620 23:26:22.093739 18964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:26:22.097378 18979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:22.098933 18964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:26:22.099107 18964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "a9e998e5513f42a681a7af15fd4b254d"
format_stamp: "Formatted at 2026-06-20 23:26:22 on dist-test-slave-5kzz"
server_key: "a833f90fd18d3c7a59f998bb708bfad1"
server_key_iv: "445a000cf37e70ddbf10e48ba5a00b64"
server_key_version: "encryptionkey@0"
I20260620 23:26:22.099295 18964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:22.119995 18964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:22.124125 18964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:22.124526 18964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:22.125540 18964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:22.126947 18964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:22.127017 18964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:22.127163 18964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:22.127223 18964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:22.161404 18964 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40945
I20260620 23:26:22.161463 19091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40945 every 8 connection(s)
I20260620 23:26:22.162797 18964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:26:22.169157 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 18964
I20260620 23:26:22.169345 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:26:22.169765 388 external_mini_cluster.cc:1428] Setting key 8219d325fba7165073d3b2915aa1d0fb
Jun 20 23:26:22 dist-test-slave-5kzz krb5kdc[18590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:22.181944 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.165384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56949 (local address 127.0.97.62:35805)
0620 23:26:22.165874 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:22.165878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:22.166659 (+ 781us) server_negotiation.cc:407] Connection header received
0620 23:26:22.168505 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:22.168512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:22.168601 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:22.168725 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:22.170375 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.171233 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.172761 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.173176 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.173312 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.176860 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:22.176865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:22.176879 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:22.176912 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:22.179410 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:22.180043 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:22.180047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:22.180049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:22.180157 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:22.180544 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:22.180547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:22.180549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:22.180792 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:22.180909 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:22.181509 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 23:26:22.181674 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
I20260620 23:26:22.183022 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.165740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35805 (local address 127.0.97.3:56949)
0620 23:26:22.166480 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:22.166528 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:22.168214 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:22.168966 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:22.168974 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:22.169576 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:22.170206 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:22.170224 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.171398 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.171402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:22.172611 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:22.172622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.173410 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.173417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:22.173695 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.174413 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:26:22.174440 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:22.176672 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:22.179528 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:22.179534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:22.179546 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:22.179947 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:22.180246 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:22.180249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:22.180251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:22.180417 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:22.180890 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:22.180897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:22.181194 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 23:26:22.181419 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 23:26:22.181805 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":290,"threads_started":1}
I20260620 23:26:22.184445 19092 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35805
I20260620 23:26:22.184777 19092 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:22.185475 19092 heartbeater.cc:511] Master 127.0.97.62:35805 requested a full tablet report, sending...
I20260620 23:26:22.186790 18635 ts_manager.cc:194] Registered new tserver with Master: a9e998e5513f42a681a7af15fd4b254d (127.0.97.3:40945)
I20260620 23:26:22.187608 18635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56949
I20260620 23:26:22.197467 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:26:22.218297 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.203822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.0.97.62:35805)
0620 23:26:22.204008 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:22.204034 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:22.204527 (+ 493us) server_negotiation.cc:407] Connection header received
0620 23:26:22.204787 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:22.204792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:22.204871 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:22.204982 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:22.206392 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.207167 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.208508 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.208937 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.209086 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.212087 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:22.212092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:22.212096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:22.212135 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:22.214935 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:22.215935 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:22.215946 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:22.215949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:22.216089 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:22.216650 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:22.216658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:22.216666 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:22.217016 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:22.217330 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:22.217725 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 23:26:22.217935 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:26:22.223081 18635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:26:22.230268 18635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:26:22.266446 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.243630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36047 (local address 127.0.0.1:60792)
0620 23:26:22.254347 (+ 10717us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:22.254462 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:22.254810 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:22.258345 (+ 3535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:22.258357 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:22.258401 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:22.259068 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:22.259085 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.261520 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.261525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:22.263737 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:22.263749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.265480 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.265486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:22.265715 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.265786 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 23:26:22.265905 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:26:22.266104 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10212,"mutex_wait_us":3,"thread_start_us":12502,"threads_started":1}
I20260620 23:26:22.267629 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.243742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.0.97.1:36047)
0620 23:26:22.257428 (+ 13686us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:22.257442 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:22.257484 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:26:22.257629 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:22.257641 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:22.258009 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:22.259358 (+ 1349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:22.259618 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.261330 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.263917 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.265210 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.266744 (+ 1534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.267034 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:22.267136 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 23:26:22.267307 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13452,"thread_start_us":13256,"threads_started":1}
I20260620 23:26:22.268200 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.243965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40945 (local address 127.0.0.1:44882)
0620 23:26:22.256502 (+ 12537us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:22.256534 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:22.256695 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:22.260393 (+ 3698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:22.260399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:22.260427 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:22.260926 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:22.260950 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.263163 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.263170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:22.265355 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:22.265368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.267415 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.267424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:22.267596 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.267628 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:26:22.267689 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:26:22.267824 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12350,"thread_start_us":11154,"threads_started":1}
I20260620 23:26:22.269106 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.244321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.0.97.3:40945)
0620 23:26:22.259535 (+ 15214us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:22.259543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:22.259581 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:26:22.259687 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:22.259694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:22.259953 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:22.260881 (+ 928us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:22.261299 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.263010 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.265570 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.267164 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.268275 (+ 1111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.268547 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:22.268617 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 23:26:22.268784 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15016,"thread_start_us":15764,"threads_started":1}
I20260620 23:26:22.270080 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.257044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34750 (local address 127.0.97.2:33827)
0620 23:26:22.260699 (+ 3655us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:22.260709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:22.261742 (+ 1033us) server_negotiation.cc:407] Connection header received
0620 23:26:22.261847 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:22.261855 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:22.262116 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:22.263056 (+ 940us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:22.263331 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.264923 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.267523 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.269020 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.269176 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.269389 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:22.269612 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 23:26:22.269774 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3492,"thread_start_us":3356,"threads_started":1}
I20260620 23:26:22.270625 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.257446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33827 (local address 127.0.0.1:34750)
0620 23:26:22.261454 (+ 4008us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:22.261474 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:22.261595 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:22.262329 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:22.262334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:22.262374 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:22.262880 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:22.262893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.265078 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.265083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:22.267267 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:22.267285 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.269236 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.269242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:22.269413 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.269451 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:26:22.269557 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:26:22.269691 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3864,"thread_start_us":3707,"threads_started":1}
I20260620 23:26:22.272614 18759 tablet_service.cc:1467] Processing CreateTablet for tablet 57ea0e5147984f97a201ec381442fe43 (DEFAULT_TABLE table=test-table [id=f4dda800823c49be8adeb540ad5840fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:22.272881 19027 tablet_service.cc:1467] Processing CreateTablet for tablet 57ea0e5147984f97a201ec381442fe43 (DEFAULT_TABLE table=test-table [id=f4dda800823c49be8adeb540ad5840fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:22.274047 18759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ea0e5147984f97a201ec381442fe43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:22.274269 19027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ea0e5147984f97a201ec381442fe43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:22.274456 18893 tablet_service.cc:1467] Processing CreateTablet for tablet 57ea0e5147984f97a201ec381442fe43 (DEFAULT_TABLE table=test-table [id=f4dda800823c49be8adeb540ad5840fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:22.275765 18893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ea0e5147984f97a201ec381442fe43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:22.285936 19111 tablet_bootstrap.cc:492] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25: Bootstrap starting.
I20260620 23:26:22.288849 19112 tablet_bootstrap.cc:492] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d: Bootstrap starting.
I20260620 23:26:22.289734 19111 tablet_bootstrap.cc:654] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:22.291109 19111 log.cc:824] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:22.292642 19112 tablet_bootstrap.cc:654] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:22.293536 19114 tablet_bootstrap.cc:492] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673: Bootstrap starting.
I20260620 23:26:22.294004 19112 log.cc:824] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:22.294476 19111 tablet_bootstrap.cc:492] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25: No bootstrap required, opened a new log
I20260620 23:26:22.294831 19111 ts_tablet_manager.cc:1397] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 23:26:22.296887 19114 tablet_bootstrap.cc:654] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:22.297281 19112 tablet_bootstrap.cc:492] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d: No bootstrap required, opened a new log
I20260620 23:26:22.297556 19112 ts_tablet_manager.cc:1397] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:26:22.297997 19114 log.cc:824] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:22.305975 19111 raft_consensus.cc:348] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bff7f8b014b14a551605474f2bc25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36047 } } peers { permanent_uuid: "b6c4342160534b75bbf4c86349b81673" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33827 } } peers { permanent_uuid: "a9e998e5513f42a681a7af15fd4b254d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40945 } }
I20260620 23:26:22.306475 19111 raft_consensus.cc:374] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:22.306581 19111 raft_consensus.cc:729] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bff7f8b014b14a551605474f2bc25, State: Initialized, Role: FOLLOWER
I20260620 23:26:22.307310 19114 tablet_bootstrap.cc:492] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673: No bootstrap required, opened a new log
I20260620 23:26:22.307302 19111 consensus_queue.cc:260] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bff7f8b014b14a551605474f2bc25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36047 } } peers { permanent_uuid: "b6c4342160534b75bbf4c86349b81673" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33827 } } peers { permanent_uuid: "a9e998e5513f42a681a7af15fd4b254d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40945 } }
I20260620 23:26:22.307657 19114 ts_tablet_manager.cc:1397] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260620 23:26:22.308729 19112 raft_consensus.cc:348] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bff7f8b014b14a551605474f2bc25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36047 } } peers { permanent_uuid: "b6c4342160534b75bbf4c86349b81673" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33827 } } peers { permanent_uuid: "a9e998e5513f42a681a7af15fd4b254d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40945 } }
I20260620 23:26:22.309306 19112 raft_consensus.cc:374] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:22.309417 19112 raft_consensus.cc:729] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e998e5513f42a681a7af15fd4b254d, State: Initialized, Role: FOLLOWER
I20260620 23:26:22.310217 19112 consensus_queue.cc:260] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bff7f8b014b14a551605474f2bc25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36047 } } peers { permanent_uuid: "b6c4342160534b75bbf4c86349b81673" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33827 } } peers { permanent_uuid: "a9e998e5513f42a681a7af15fd4b254d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40945 } }
I20260620 23:26:22.314218 19111 ts_tablet_manager.cc:1428] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25: Time spent starting tablet: real 0.019s user 0.016s sys 0.000s
I20260620 23:26:22.315344 19092 heartbeater.cc:503] Master 127.0.97.62:35805 was elected leader, sending a full tablet report...
I20260620 23:26:22.316035 19112 ts_tablet_manager.cc:1428] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d: Time spent starting tablet: real 0.018s user 0.012s sys 0.008s
I20260620 23:26:22.318118 19114 raft_consensus.cc:348] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bff7f8b014b14a551605474f2bc25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36047 } } peers { permanent_uuid: "b6c4342160534b75bbf4c86349b81673" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33827 } } peers { permanent_uuid: "a9e998e5513f42a681a7af15fd4b254d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40945 } }
I20260620 23:26:22.318761 19114 raft_consensus.cc:374] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:22.318876 19114 raft_consensus.cc:729] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c4342160534b75bbf4c86349b81673, State: Initialized, Role: FOLLOWER
I20260620 23:26:22.319589 19114 consensus_queue.cc:260] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bff7f8b014b14a551605474f2bc25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36047 } } peers { permanent_uuid: "b6c4342160534b75bbf4c86349b81673" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33827 } } peers { permanent_uuid: "a9e998e5513f42a681a7af15fd4b254d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40945 } }
I20260620 23:26:22.320755 18958 heartbeater.cc:503] Master 127.0.97.62:35805 was elected leader, sending a full tablet report...
I20260620 23:26:22.321386 19114 ts_tablet_manager.cc:1428] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
W20260620 23:26:22.384478 18959 tablet.cc:2369] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:26:22.414475 19093 tablet.cc:2369] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:26:22.514218 19119 raft_consensus.cc:482] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:26:22.514525 19119 raft_consensus.cc:504] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bff7f8b014b14a551605474f2bc25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36047 } } peers { permanent_uuid: "b6c4342160534b75bbf4c86349b81673" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33827 } } peers { permanent_uuid: "a9e998e5513f42a681a7af15fd4b254d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40945 } }
I20260620 23:26:22.516362 19119 leader_election.cc:288] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b9bff7f8b014b14a551605474f2bc25 (127.0.97.1:36047), a9e998e5513f42a681a7af15fd4b254d (127.0.97.3:40945)
I20260620 23:26:22.526242 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.516876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37685 (local address 127.0.97.1:36047)
0620 23:26:22.517114 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:22.517120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:22.517963 (+ 843us) server_negotiation.cc:407] Connection header received
0620 23:26:22.518143 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:22.518149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:22.518233 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:22.518352 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:22.519232 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.520403 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.523635 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.524964 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.525104 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.525216 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:22.525834 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 23:26:22.525978 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 23:26:22.526235 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.516911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58075 (local address 127.0.97.3:40945)
0620 23:26:22.517113 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:22.517120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:22.517964 (+ 844us) server_negotiation.cc:407] Connection header received
0620 23:26:22.518143 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:22.518149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:22.518241 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:22.518370 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:22.519229 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.520419 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.523637 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.524951 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.525104 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.525216 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:22.525834 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 23:26:22.525978 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:26:22.526464 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.516744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40945 (local address 127.0.97.2:58075)
0620 23:26:22.517819 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:22.517834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:22.517992 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:22.518398 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:22.518403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:22.518458 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:22.519010 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:22.519028 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.520548 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.520553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:22.523432 (+ 2879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:22.523449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.525260 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.525268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:22.525603 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.525651 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 23:26:22.525763 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 23:26:22.525955 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":327,"spinlock_wait_cycles":26880,"thread_start_us":552,"threads_started":1}
I20260620 23:26:22.526907 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.516708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36047 (local address 127.0.97.2:37685)
0620 23:26:22.517791 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:22.517817 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:22.517992 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:22.518398 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:22.518403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:22.518457 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:22.519073 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:22.519084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.520548 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.520553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:22.523432 (+ 2879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:22.523449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.525300 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.525305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:22.525597 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.525659 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 23:26:22.525763 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:26:22.525956 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":287,"spinlock_wait_cycles":64640,"thread_start_us":643,"threads_started":1}
I20260620 23:26:22.527596 19047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ea0e5147984f97a201ec381442fe43" candidate_uuid: "b6c4342160534b75bbf4c86349b81673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e998e5513f42a681a7af15fd4b254d" is_pre_election: true
I20260620 23:26:22.527848 18779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ea0e5147984f97a201ec381442fe43" candidate_uuid: "b6c4342160534b75bbf4c86349b81673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b9bff7f8b014b14a551605474f2bc25" is_pre_election: true
I20260620 23:26:22.528112 19047 raft_consensus.cc:2454] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c4342160534b75bbf4c86349b81673 in term 0.
I20260620 23:26:22.528425 18779 raft_consensus.cc:2454] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c4342160534b75bbf4c86349b81673 in term 0.
I20260620 23:26:22.528842 18848 leader_election.cc:302] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e998e5513f42a681a7af15fd4b254d, b6c4342160534b75bbf4c86349b81673; no voters:
I20260620 23:26:22.529217 19119 raft_consensus.cc:2781] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:26:22.529382 19119 raft_consensus.cc:482] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:26:22.529505 19119 raft_consensus.cc:3037] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:22.531778 19119 raft_consensus.cc:504] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bff7f8b014b14a551605474f2bc25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36047 } } peers { permanent_uuid: "b6c4342160534b75bbf4c86349b81673" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33827 } } peers { permanent_uuid: "a9e998e5513f42a681a7af15fd4b254d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40945 } }
I20260620 23:26:22.532400 19119 leader_election.cc:288] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [CANDIDATE]: Term 1 election: Requested vote from peers 4b9bff7f8b014b14a551605474f2bc25 (127.0.97.1:36047), a9e998e5513f42a681a7af15fd4b254d (127.0.97.3:40945)
I20260620 23:26:22.532809 18779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ea0e5147984f97a201ec381442fe43" candidate_uuid: "b6c4342160534b75bbf4c86349b81673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b9bff7f8b014b14a551605474f2bc25"
I20260620 23:26:22.532888 19047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ea0e5147984f97a201ec381442fe43" candidate_uuid: "b6c4342160534b75bbf4c86349b81673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e998e5513f42a681a7af15fd4b254d"
I20260620 23:26:22.533041 18779 raft_consensus.cc:3037] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:22.533151 19047 raft_consensus.cc:3037] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:22.535439 19047 raft_consensus.cc:2454] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c4342160534b75bbf4c86349b81673 in term 1.
I20260620 23:26:22.535444 18779 raft_consensus.cc:2454] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c4342160534b75bbf4c86349b81673 in term 1.
I20260620 23:26:22.535930 18848 leader_election.cc:302] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e998e5513f42a681a7af15fd4b254d, b6c4342160534b75bbf4c86349b81673; no voters:
I20260620 23:26:22.536224 19119 raft_consensus.cc:2781] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:22.536737 19119 raft_consensus.cc:686] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 1 LEADER]: Becoming Leader. State: Replica: b6c4342160534b75bbf4c86349b81673, State: Running, Role: LEADER
I20260620 23:26:22.537217 19119 consensus_queue.cc:237] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bff7f8b014b14a551605474f2bc25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36047 } } peers { permanent_uuid: "b6c4342160534b75bbf4c86349b81673" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33827 } } peers { permanent_uuid: "a9e998e5513f42a681a7af15fd4b254d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40945 } }
I20260620 23:26:22.541842 18635 catalog_manager.cc:5495] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6c4342160534b75bbf4c86349b81673 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b6c4342160534b75bbf4c86349b81673" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bff7f8b014b14a551605474f2bc25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6c4342160534b75bbf4c86349b81673" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e998e5513f42a681a7af15fd4b254d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40945 } health_report { overall_health: UNKNOWN } } }
W20260620 23:26:22.546743 18825 tablet.cc:2369] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:26:22.577031 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.571515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.0.97.2:33827)
0620 23:26:22.571697 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:22.571701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:22.571719 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:26:22.571805 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:22.571808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:22.571886 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:22.572016 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:26:22.572497 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.573293 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.574888 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.575379 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.575485 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.575976 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:26:22.576519 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:26:22.576658 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:22.576776 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 23:26:22.576819 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:26:22.585574 19047 raft_consensus.cc:1261] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d [term 1 FOLLOWER]: Refusing update from remote peer b6c4342160534b75bbf4c86349b81673: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:26:22.585574 18779 raft_consensus.cc:1261] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25 [term 1 FOLLOWER]: Refusing update from remote peer b6c4342160534b75bbf4c86349b81673: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:26:22.586755 19122 consensus_queue.cc:1035] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e998e5513f42a681a7af15fd4b254d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:26:22.587179 19119 consensus_queue.cc:1035] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bff7f8b014b14a551605474f2bc25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:26:22.603658 19127 mvcc.cc:205] Tried to move back new op lower bound from 7299063736660033536 to 7299063736476811264. Current Snapshot: MvccSnapshot[applied={T|T < 7299063736660033536}]
I20260620 23:26:22.612524 18635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:26:22.612818 18635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:26:22.616714 18635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d545282b5494399b888d92e2a25d78a: Sending DeleteTablet for 3 replicas of tablet 57ea0e5147984f97a201ec381442fe43
I20260620 23:26:22.617938 18759 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ea0e5147984f97a201ec381442fe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60792
I20260620 23:26:22.617947 18893 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ea0e5147984f97a201ec381442fe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34750
I20260620 23:26:22.619257 19027 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ea0e5147984f97a201ec381442fe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44882
I20260620 23:26:22.621224 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 18696
I20260620 23:26:22.637457 19136 tablet_replica.cc:330] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d: stopping tablet replica
I20260620 23:26:22.637945 19136 raft_consensus.cc:2229] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:26:22.638482 19136 raft_consensus.cc:2258] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:22.639669 19135 tablet_replica.cc:330] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673: stopping tablet replica
I20260620 23:26:22.640781 19135 raft_consensus.cc:2229] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:26:22.642105 19135 raft_consensus.cc:2258] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:22.642854 19134 tablet_replica.cc:330] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25: stopping tablet replica
I20260620 23:26:22.643252 19134 raft_consensus.cc:2229] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:26:22.643725 19134 raft_consensus.cc:2258] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:22.644881 19135 ts_tablet_manager.cc:1905] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:22.645147 19136 ts_tablet_manager.cc:1905] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 23:26:22 dist-test-slave-5kzz krb5kdc[18590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:26:22.652917 19135 ts_tablet_manager.cc:1918] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:22.654095 19134 ts_tablet_manager.cc:1905] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:22.654258 19135 log.cc:1196] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/57ea0e5147984f97a201ec381442fe43
I20260620 23:26:22.654829 19135 ts_tablet_manager.cc:1939] T 57ea0e5147984f97a201ec381442fe43 P b6c4342160534b75bbf4c86349b81673: Deleting consensus metadata
I20260620 23:26:22.656775 18619 catalog_manager.cc:4841] TS b6c4342160534b75bbf4c86349b81673 (127.0.97.2:33827): tablet 57ea0e5147984f97a201ec381442fe43 (table test-table [id=f4dda800823c49be8adeb540ad5840fd]) successfully deleted
I20260620 23:26:22.660682 19136 ts_tablet_manager.cc:1918] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:22.660962 19136 log.cc:1196] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/57ea0e5147984f97a201ec381442fe43
I20260620 23:26:22.661578 19136 ts_tablet_manager.cc:1939] T 57ea0e5147984f97a201ec381442fe43 P a9e998e5513f42a681a7af15fd4b254d: Deleting consensus metadata
I20260620 23:26:22.662369 19134 ts_tablet_manager.cc:1918] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:22.662554 19134 log.cc:1196] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/57ea0e5147984f97a201ec381442fe43
I20260620 23:26:22.663522 19134 ts_tablet_manager.cc:1939] T 57ea0e5147984f97a201ec381442fe43 P 4b9bff7f8b014b14a551605474f2bc25: Deleting consensus metadata
I20260620 23:26:22.663781 18621 catalog_manager.cc:4841] TS a9e998e5513f42a681a7af15fd4b254d (127.0.97.3:40945): tablet 57ea0e5147984f97a201ec381442fe43 (table test-table [id=f4dda800823c49be8adeb540ad5840fd]) successfully deleted
I20260620 23:26:22.665400 18619 catalog_manager.cc:4841] TS 4b9bff7f8b014b14a551605474f2bc25 (127.0.97.1:36047): tablet 57ea0e5147984f97a201ec381442fe43 (table test-table [id=f4dda800823c49be8adeb540ad5840fd]) successfully deleted
I20260620 23:26:22.668438 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.622570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.0.97.1:36047)
0620 23:26:22.622775 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:22.622782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:22.634553 (+ 11771us) server_negotiation.cc:407] Connection header received
0620 23:26:22.635034 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:22.635040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:22.635140 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:22.635223 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:22.638819 (+ 3596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.639974 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.646830 (+ 6856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.647548 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.647655 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.660685 (+ 13030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:22.660706 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:22.660727 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:22.660774 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:22.665056 (+ 4282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:22.666086 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:22.666095 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:22.666097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:22.666232 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:22.666738 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:22.666749 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:22.666752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:22.667113 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:22.667332 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:22.667789 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 23:26:22.668058 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:26:22.669126 18819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:22.761761 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 18696
I20260620 23:26:22.780579 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 18830
Jun 20 23:26:22 dist-test-slave-5kzz krb5kdc[18590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:26:22.800148 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.781547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.0.97.2:33827)
0620 23:26:22.781759 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:22.781766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:22.781788 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:26:22.781954 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:22.781960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:22.782062 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:22.782177 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:22.783581 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.785044 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.786770 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.787480 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.787632 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.792665 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:22.792683 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:22.792701 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:22.792755 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:22.796792 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:22.797681 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:22.797692 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:22.797695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:22.797868 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:22.798280 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:22.798285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:22.798289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:22.798771 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:22.799114 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:22.799431 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 23:26:22.799658 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:26:22.801047 18953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:22.916688 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 18830
I20260620 23:26:22.936801 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 18964
Jun 20 23:26:22 dist-test-slave-5kzz krb5kdc[18590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:26:22.952255 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:22.937595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34660 (local address 127.0.97.3:40945)
0620 23:26:22.937766 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:22.937770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:22.937791 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:26:22.937979 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:22.937984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:22.938056 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:22.938143 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:22.939641 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.940383 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.941688 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:22.942090 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:22.942190 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:22.946568 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:22.946584 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:22.946600 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:22.946637 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:22.949531 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:22.950261 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:22.950269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:22.950271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:22.950418 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:22.950882 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:22.950887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:22.950889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:22.951283 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:22.951507 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:22.951819 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 23:26:22.951997 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:26:22.952922 19087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:23.043903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 18964
I20260620 23:26:23.060815 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 18603
I20260620 23:26:23.062005 18665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:23.174252 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 18603
2026-06-20T23:26:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4629 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T23:26:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:23Z Disabled control of system clock
I20260620 23:26:23.255383 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36041
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:26:23.606184 19145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:23.606515 19145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:23.606658 19145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:23.614277 19145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:23.614418 19145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:23.614491 19145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:23.614578 19145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:23.614671 19145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:23.624310 19145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:23.625689 19145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:23.627732 19145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:23.639691 19151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:23.639770 19150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:23.639748 19153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:23.641135 19145 server_base.cc:1034] running on GCE node
I20260620 23:26:23.641947 19145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:23.643476 19145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:23.644706 19145 hybrid_clock.cc:648] HybridClock initialized: now 1781997983644659 us; error 45 us; skew 500 ppm
I20260620 23:26:23.645067 19145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:26:23.647481 19145 webserver.cc:460] Webserver started at http://127.0.97.62:41915/ using document root <none> and password file <none>
I20260620 23:26:23.648180 19145 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:23.648312 19145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:23.648641 19145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:23.651722 19145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "e6f8a3a8683545aab09892d110aeb51f"
format_stamp: "Formatted at 2026-06-20 23:26:23 on dist-test-slave-5kzz"
server_key: "a385d36d5c853baeb087341890738921"
server_key_iv: "6f5ff3735cf8e69cb05e9aae614a9d03"
server_key_version: "encryptionkey@0"
I20260620 23:26:23.652559 19145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "e6f8a3a8683545aab09892d110aeb51f"
format_stamp: "Formatted at 2026-06-20 23:26:23 on dist-test-slave-5kzz"
server_key: "a385d36d5c853baeb087341890738921"
server_key_iv: "6f5ff3735cf8e69cb05e9aae614a9d03"
server_key_version: "encryptionkey@0"
I20260620 23:26:23.658105 19145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:26:23.662010 19159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:23.663551 19145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:26:23.663755 19145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "e6f8a3a8683545aab09892d110aeb51f"
format_stamp: "Formatted at 2026-06-20 23:26:23 on dist-test-slave-5kzz"
server_key: "a385d36d5c853baeb087341890738921"
server_key_iv: "6f5ff3735cf8e69cb05e9aae614a9d03"
server_key_version: "encryptionkey@0"
I20260620 23:26:23.663995 19145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:23.691334 19145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:23.692274 19145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:23.692587 19145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:23.715335 19145 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36041
I20260620 23:26:23.715265 19210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36041 every 8 connection(s)
I20260620 23:26:23.717476 19145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:23.720800 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 19145
I20260620 23:26:23.721035 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:23.721756 388 external_mini_cluster.cc:1428] Setting key 89aff94776af11849aad1e32ba59a30b
I20260620 23:26:23.724745 19211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:23.743657 19211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f: Bootstrap starting.
I20260620 23:26:23.747148 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:23.724273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.0.97.62:36041)
0620 23:26:23.730168 (+ 5895us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:23.730216 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:23.730334 (+ 118us) server_negotiation.cc:407] Connection header received
0620 23:26:23.733298 (+ 2964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:23.733351 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:23.734312 (+ 961us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:23.734979 (+ 667us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:23.736029 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:23.737625 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:23.740148 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:23.741512 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:23.741727 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:23.743370 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:23.743383 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:23.743409 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:23.744322 (+ 913us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:23.744456 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:23.744495 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:23.744651 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:23.744990 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 23:26:23.745453 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5368,"thread_start_us":5065,"threads_started":1}
I20260620 23:26:23.747648 19211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:23.750654 19211 log.cc:824] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:23.755786 19211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f: No bootstrap required, opened a new log
I20260620 23:26:23.764784 19211 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f8a3a8683545aab09892d110aeb51f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } }
I20260620 23:26:23.765326 19211 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:23.765450 19211 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f8a3a8683545aab09892d110aeb51f, State: Initialized, Role: FOLLOWER
I20260620 23:26:23.766052 19211 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f8a3a8683545aab09892d110aeb51f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } }
I20260620 23:26:23.766328 19211 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:23.766477 19211 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:23.766669 19211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:23.769508 19211 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f8a3a8683545aab09892d110aeb51f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } }
I20260620 23:26:23.770152 19211 leader_election.cc:302] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6f8a3a8683545aab09892d110aeb51f; no voters:
I20260620 23:26:23.770924 19211 leader_election.cc:288] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:23.771096 19216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:23.771822 19216 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [term 1 LEADER]: Becoming Leader. State: Replica: e6f8a3a8683545aab09892d110aeb51f, State: Running, Role: LEADER
I20260620 23:26:23.772424 19216 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f8a3a8683545aab09892d110aeb51f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } }
I20260620 23:26:23.773379 19211 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:23.775484 19217 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6f8a3a8683545aab09892d110aeb51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f8a3a8683545aab09892d110aeb51f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } }
I20260620 23:26:23.775887 19217 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:23.776592 19218 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6f8a3a8683545aab09892d110aeb51f. Latest consensus state: current_term: 1 leader_uuid: "e6f8a3a8683545aab09892d110aeb51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f8a3a8683545aab09892d110aeb51f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } }
I20260620 23:26:23.776849 19218 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:23.777865 19222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:23.785388 19222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:23.795763 19222 catalog_manager.cc:1347] Generated new cluster ID: 58e3f6488e5b4ee78caf2e19adfe776c
I20260620 23:26:23.795915 19222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:23.838304 19222 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:23.839818 19222 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:23.859191 19222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f: Generated new TSK 0
I20260620 23:26:23.860176 19222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:23.867811 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36041
--builtin_ntp_servers=127.0.97.20:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:26:24.225909 19235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:24.226217 19235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:24.226315 19235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:24.233816 19235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:24.233942 19235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:24.234055 19235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:26:24.243947 19235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:24.245555 19235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:24.247495 19235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:24.261809 19240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:24.261837 19243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:24.261773 19241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:24.262555 19235 server_base.cc:1034] running on GCE node
I20260620 23:26:24.263278 19235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:24.264286 19235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:24.265504 19235 hybrid_clock.cc:648] HybridClock initialized: now 1781997984265458 us; error 43 us; skew 500 ppm
I20260620 23:26:24.265827 19235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:26:24.268592 19235 webserver.cc:460] Webserver started at http://127.0.97.1:39161/ using document root <none> and password file <none>
I20260620 23:26:24.269408 19235 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:24.269511 19235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:24.269801 19235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:24.273032 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "c570c8606ab844a881ecda91c03cd6f2"
format_stamp: "Formatted at 2026-06-20 23:26:24 on dist-test-slave-5kzz"
server_key: "f714eca6b6d3f1281c20b87733bb05b9"
server_key_iv: "229cfcfeb5bc4623aa0643238c9ded90"
server_key_version: "encryptionkey@0"
I20260620 23:26:24.274055 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c570c8606ab844a881ecda91c03cd6f2"
format_stamp: "Formatted at 2026-06-20 23:26:24 on dist-test-slave-5kzz"
server_key: "f714eca6b6d3f1281c20b87733bb05b9"
server_key_iv: "229cfcfeb5bc4623aa0643238c9ded90"
server_key_version: "encryptionkey@0"
I20260620 23:26:24.279942 19235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:26:24.283977 19249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:24.285492 19235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:26:24.285645 19235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "c570c8606ab844a881ecda91c03cd6f2"
format_stamp: "Formatted at 2026-06-20 23:26:24 on dist-test-slave-5kzz"
server_key: "f714eca6b6d3f1281c20b87733bb05b9"
server_key_iv: "229cfcfeb5bc4623aa0643238c9ded90"
server_key_version: "encryptionkey@0"
I20260620 23:26:24.285843 19235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:24.301824 19235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:24.302634 19235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:24.302964 19235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:24.303946 19235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:24.305440 19235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:24.305526 19235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:24.305603 19235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:24.305657 19235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:24.341872 19235 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42939
I20260620 23:26:24.341931 19361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42939 every 8 connection(s)
I20260620 23:26:24.343384 19235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:26:24.349599 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 19235
I20260620 23:26:24.349779 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:26:24.350356 388 external_mini_cluster.cc:1428] Setting key dd3ec68c9cf9db02360a925d19912f93
I20260620 23:26:24.353221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36041
--builtin_ntp_servers=127.0.97.20:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:26:24.357184 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:24.346323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48195 (local address 127.0.97.62:36041)
0620 23:26:24.346883 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:24.346887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:24.347959 (+ 1072us) server_negotiation.cc:407] Connection header received
0620 23:26:24.350121 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:24.350126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:24.350210 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:24.350308 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:24.351766 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:24.352542 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:24.354253 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:24.354660 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:24.354859 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:24.356309 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:24.356314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:24.356318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:24.356338 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:24.356378 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:24.356391 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:24.356491 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:24.356715 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 23:26:24.356827 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":451,"thread_start_us":315,"threads_started":1}
I20260620 23:26:24.358249 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:24.346814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36041 (local address 127.0.97.1:48195)
0620 23:26:24.347769 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:24.347816 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:24.349793 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:24.350607 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:24.350619 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:24.350776 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:24.351593 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:24.351612 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:24.352715 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:24.352720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:24.354091 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:24.354103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:24.354944 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:24.354954 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:24.355218 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:24.355971 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:26:24.355974 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:24.356014 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:26:24.356097 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:24.356489 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:24.356502 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:24.356512 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:26:24.356706 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 23:26:24.357128 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":293,"threads_started":1}
I20260620 23:26:24.360071 19362 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36041
I20260620 23:26:24.360425 19362 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:24.361533 19362 heartbeater.cc:511] Master 127.0.97.62:36041 requested a full tablet report, sending...
I20260620 23:26:24.364280 19176 ts_manager.cc:194] Registered new tserver with Master: c570c8606ab844a881ecda91c03cd6f2 (127.0.97.1:42939)
I20260620 23:26:24.366436 19176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:48195
W20260620 23:26:24.697400 19366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:24.697679 19366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:24.697765 19366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:24.705117 19366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:24.705225 19366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:24.705338 19366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:26:24.714437 19366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:24.715759 19366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:24.717511 19366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:24.730850 19371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:24.730856 19372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:24.731499 19374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:24.732159 19366 server_base.cc:1034] running on GCE node
I20260620 23:26:24.732869 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:24.733901 19366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:24.735136 19366 hybrid_clock.cc:648] HybridClock initialized: now 1781997984735057 us; error 57 us; skew 500 ppm
I20260620 23:26:24.735513 19366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:26:24.737999 19366 webserver.cc:460] Webserver started at http://127.0.97.2:36883/ using document root <none> and password file <none>
I20260620 23:26:24.738848 19366 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:24.738948 19366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:24.739326 19366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:24.742295 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "d35d135641f340528bbec4b287b3c29b"
format_stamp: "Formatted at 2026-06-20 23:26:24 on dist-test-slave-5kzz"
server_key: "1240f4c57a7bdc9426cb542aa0db0dc8"
server_key_iv: "f133cedca64e6b1703b8d4f0993f591d"
server_key_version: "encryptionkey@0"
I20260620 23:26:24.743178 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d35d135641f340528bbec4b287b3c29b"
format_stamp: "Formatted at 2026-06-20 23:26:24 on dist-test-slave-5kzz"
server_key: "1240f4c57a7bdc9426cb542aa0db0dc8"
server_key_iv: "f133cedca64e6b1703b8d4f0993f591d"
server_key_version: "encryptionkey@0"
I20260620 23:26:24.748458 19366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:26:24.752094 19380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:24.753542 19366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:26:24.753736 19366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "d35d135641f340528bbec4b287b3c29b"
format_stamp: "Formatted at 2026-06-20 23:26:24 on dist-test-slave-5kzz"
server_key: "1240f4c57a7bdc9426cb542aa0db0dc8"
server_key_iv: "f133cedca64e6b1703b8d4f0993f591d"
server_key_version: "encryptionkey@0"
I20260620 23:26:24.753969 19366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:24.771898 19366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:24.772754 19366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:24.773056 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:24.774181 19366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:24.775836 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:24.775966 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:24.776103 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:24.776201 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:24.810218 19366 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43867
I20260620 23:26:24.810266 19492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43867 every 8 connection(s)
I20260620 23:26:24.811560 19366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:26:24.812150 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 19366
I20260620 23:26:24.812376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:26:24.812963 388 external_mini_cluster.cc:1428] Setting key 386adeef5051f6be0ce17e008af127e2
I20260620 23:26:24.816413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36041
--builtin_ntp_servers=127.0.97.20:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:26:24.828187 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:24.814837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42337 (local address 127.0.97.62:36041)
0620 23:26:24.815015 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:24.815019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:24.816754 (+ 1735us) server_negotiation.cc:407] Connection header received
0620 23:26:24.819686 (+ 2932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:24.819694 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:24.819795 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:24.819899 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:24.821316 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:24.822321 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:24.824887 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:24.825436 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:24.825533 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:24.827234 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:24.827244 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:24.827252 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:24.827280 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:24.827332 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:24.827350 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:24.827443 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:24.827661 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:26:24.827778 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:26:24.829054 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:24.815493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36041 (local address 127.0.97.2:42337)
0620 23:26:24.816593 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:24.816649 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:24.819371 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:24.820182 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:24.820196 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:24.820318 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:24.821142 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:24.821170 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:24.822485 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:24.822493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:24.824713 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:24.824732 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:24.825672 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:24.825682 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:24.826009 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:24.826900 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:26:24.826903 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:24.826945 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:26:24.827026 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:24.827450 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:24.827458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:24.827464 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:26:24.827605 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 23:26:24.827915 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":328,"threads_started":1}
I20260620 23:26:24.830612 19493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36041
I20260620 23:26:24.830941 19493 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:24.831707 19493 heartbeater.cc:511] Master 127.0.97.62:36041 requested a full tablet report, sending...
I20260620 23:26:24.833238 19176 ts_manager.cc:194] Registered new tserver with Master: d35d135641f340528bbec4b287b3c29b (127.0.97.2:43867)
I20260620 23:26:24.834481 19176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:42337
W20260620 23:26:25.156185 19496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:25.156466 19496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:25.156554 19496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:25.163895 19496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:25.164005 19496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:25.164110 19496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:26:25.173359 19496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:25.174798 19496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:25.176564 19496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:25.190088 19504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:25.190169 19502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:25.190124 19501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:25.195560 19496 server_base.cc:1034] running on GCE node
I20260620 23:26:25.196255 19496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:25.197329 19496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:25.198602 19496 hybrid_clock.cc:648] HybridClock initialized: now 1781997985198554 us; error 42 us; skew 500 ppm
I20260620 23:26:25.199015 19496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:26:25.201603 19496 webserver.cc:460] Webserver started at http://127.0.97.3:43731/ using document root <none> and password file <none>
I20260620 23:26:25.202365 19496 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:25.202507 19496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:25.202922 19496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:25.205996 19496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "1c84c0506b8c4e068fb33de4c37264ab"
format_stamp: "Formatted at 2026-06-20 23:26:25 on dist-test-slave-5kzz"
server_key: "953af6027896133422338b4bc88079cd"
server_key_iv: "338ef6a511d720cec63837a298c1e07f"
server_key_version: "encryptionkey@0"
I20260620 23:26:25.206898 19496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1c84c0506b8c4e068fb33de4c37264ab"
format_stamp: "Formatted at 2026-06-20 23:26:25 on dist-test-slave-5kzz"
server_key: "953af6027896133422338b4bc88079cd"
server_key_iv: "338ef6a511d720cec63837a298c1e07f"
server_key_version: "encryptionkey@0"
I20260620 23:26:25.212379 19496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:26:25.216197 19510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:25.217662 19496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:26:25.217866 19496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "1c84c0506b8c4e068fb33de4c37264ab"
format_stamp: "Formatted at 2026-06-20 23:26:25 on dist-test-slave-5kzz"
server_key: "953af6027896133422338b4bc88079cd"
server_key_iv: "338ef6a511d720cec63837a298c1e07f"
server_key_version: "encryptionkey@0"
I20260620 23:26:25.218109 19496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:25.240399 19496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:25.241395 19496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:25.241726 19496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:25.242815 19496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:25.244431 19496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:25.244572 19496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:25.244720 19496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:25.244809 19496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:25.279526 19496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34009
I20260620 23:26:25.279614 19622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34009 every 8 connection(s)
I20260620 23:26:25.280843 19496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:26:25.285977 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 19496
I20260620 23:26:25.286142 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:26:25.286700 388 external_mini_cluster.cc:1428] Setting key bf10dc2852bc391e0819a161e2aa53e7
I20260620 23:26:25.294643 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.283404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35789 (local address 127.0.97.62:36041)
0620 23:26:25.283585 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:25.283590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:25.284996 (+ 1406us) server_negotiation.cc:407] Connection header received
0620 23:26:25.287680 (+ 2684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:25.287684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:25.287810 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:25.287918 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:25.289295 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.290229 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.291938 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.292375 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.292482 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.293891 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:25.293896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:25.293899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:25.293918 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:25.293955 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:25.293967 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:25.294046 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:25.294298 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 23:26:25.294388 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:26:25.295670 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.283917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36041 (local address 127.0.97.3:35789)
0620 23:26:25.284835 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.284880 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:25.287348 (+ 2468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:25.288195 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:25.288207 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:25.288351 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:25.289123 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:25.289140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.290397 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.290401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:25.291776 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:25.291788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.292566 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.292571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:25.292787 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.293544 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:26:25.293546 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:25.293589 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:26:25.293676 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:25.294059 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:25.294066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:25.294072 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:26:25.294219 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 23:26:25.294535 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":271,"threads_started":1}
I20260620 23:26:25.297250 19623 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36041
I20260620 23:26:25.297580 19623 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:25.298318 19623 heartbeater.cc:511] Master 127.0.97.62:36041 requested a full tablet report, sending...
I20260620 23:26:25.299875 19176 ts_manager.cc:194] Registered new tserver with Master: 1c84c0506b8c4e068fb33de4c37264ab (127.0.97.3:34009)
I20260620 23:26:25.300855 19176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:35789
I20260620 23:26:25.302335 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:26:25.312878 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.306989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.0.97.62:36041)
0620 23:26:25.307165 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:25.307169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:25.307526 (+ 357us) server_negotiation.cc:407] Connection header received
0620 23:26:25.307646 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:25.307650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:25.307731 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:25.307824 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:25.308299 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.309003 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.310393 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.310901 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.310996 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.312100 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:25.312105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:25.312168 (+ 63us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:25.312190 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:25.312227 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:25.312242 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:25.312380 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:25.312583 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 23:26:25.312633 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:26:25.316797 19176 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:26:25.323943 19176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:26:25.347908 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.336363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43867 (local address 127.0.0.1:38672)
0620 23:26:25.339042 (+ 2679us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.339071 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:25.339173 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:25.340282 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:25.340287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:25.340313 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:25.340757 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:25.340773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.342578 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.342585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:25.344903 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:25.344918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.346931 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.346938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:25.347152 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.347211 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 23:26:25.347330 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:26:25.347520 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2558,"mutex_wait_us":25,"thread_start_us":3215,"threads_started":1}
I20260620 23:26:25.348075 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.336310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42939 (local address 127.0.0.1:57162)
0620 23:26:25.337075 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.337122 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:25.337302 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:25.338958 (+ 1656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:25.338967 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:25.338998 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:25.339480 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:25.339492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.341731 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.341739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:25.344623 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:25.344638 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.346700 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.346709 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:25.346949 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.347491 (+ 542us) client_negotiation.cc:769] Sending connection context
0620 23:26:25.347576 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:26:25.347708 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":992,"thread_start_us":796,"threads_started":1}
I20260620 23:26:25.348841 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.336907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.0.97.1:42939)
0620 23:26:25.338217 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:25.338227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:25.338260 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:26:25.338380 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:25.338388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:25.338668 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:25.339719 (+ 1051us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:25.339946 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.341563 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.344903 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.346475 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.347059 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.347363 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:25.348201 (+ 838us) server_negotiation.cc:299] Negotiation successful
0620 23:26:25.348412 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1136,"thread_start_us":1019,"threads_started":1}
I20260620 23:26:25.350015 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.337001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.0.97.2:43867)
0620 23:26:25.339682 (+ 2681us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:25.339690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:25.339716 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:26:25.339825 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:25.339832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:25.340078 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:25.340274 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:25.340962 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.342408 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.345127 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.346695 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.349142 (+ 2447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.349405 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:25.349476 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 23:26:25.349643 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2500,"thread_start_us":2375,"threads_started":1}
I20260620 23:26:25.350353 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.336834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34009 (local address 127.0.0.1:40326)
0620 23:26:25.339811 (+ 2977us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.339825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:25.339969 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:25.340633 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:25.340638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:25.340663 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:25.341030 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:25.341042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.343107 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.343112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:25.346177 (+ 3065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:25.346194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.349483 (+ 3289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.349490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:25.349659 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.349691 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:26:25.349832 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 23:26:25.349971 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2866,"mutex_wait_us":3,"thread_start_us":1940,"threads_started":1}
I20260620 23:26:25.350425 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.337456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.0.97.3:34009)
0620 23:26:25.338564 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:25.338571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:25.339936 (+ 1365us) server_negotiation.cc:407] Connection header received
0620 23:26:25.340174 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:25.340180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:25.340428 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:25.340627 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:25.341229 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.342916 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.346398 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.349269 (+ 2871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.349412 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.349655 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:25.349882 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 23:26:25.350053 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":794,"threads_started":1}
I20260620 23:26:25.353107 19297 tablet_service.cc:1467] Processing CreateTablet for tablet c6854619733d4eae8dfb740fd4b4ad5c (DEFAULT_TABLE table=test-table [id=0fd23e11c402475492e70493c5592cf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:25.353626 19428 tablet_service.cc:1467] Processing CreateTablet for tablet c6854619733d4eae8dfb740fd4b4ad5c (DEFAULT_TABLE table=test-table [id=0fd23e11c402475492e70493c5592cf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:25.353705 19558 tablet_service.cc:1467] Processing CreateTablet for tablet c6854619733d4eae8dfb740fd4b4ad5c (DEFAULT_TABLE table=test-table [id=0fd23e11c402475492e70493c5592cf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:25.354732 19297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6854619733d4eae8dfb740fd4b4ad5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:25.355000 19428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6854619733d4eae8dfb740fd4b4ad5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:25.355023 19558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6854619733d4eae8dfb740fd4b4ad5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:25.367151 19641 tablet_bootstrap.cc:492] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab: Bootstrap starting.
I20260620 23:26:25.370108 19362 heartbeater.cc:503] Master 127.0.97.62:36041 was elected leader, sending a full tablet report...
I20260620 23:26:25.370180 19642 tablet_bootstrap.cc:492] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2: Bootstrap starting.
I20260620 23:26:25.371748 19643 tablet_bootstrap.cc:492] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b: Bootstrap starting.
I20260620 23:26:25.373260 19641 tablet_bootstrap.cc:654] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:25.373929 19642 tablet_bootstrap.cc:654] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:25.374984 19641 log.cc:824] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:25.375074 19642 log.cc:824] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:25.375263 19643 tablet_bootstrap.cc:654] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:25.376475 19643 log.cc:824] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:25.378175 19641 tablet_bootstrap.cc:492] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab: No bootstrap required, opened a new log
I20260620 23:26:25.378224 19642 tablet_bootstrap.cc:492] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2: No bootstrap required, opened a new log
I20260620 23:26:25.378455 19641 ts_tablet_manager.cc:1397] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260620 23:26:25.378474 19642 ts_tablet_manager.cc:1397] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 23:26:25.379426 19643 tablet_bootstrap.cc:492] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b: No bootstrap required, opened a new log
I20260620 23:26:25.379689 19643 ts_tablet_manager.cc:1397] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 23:26:25.386914 19641 raft_consensus.cc:348] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35d135641f340528bbec4b287b3c29b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43867 } } peers { permanent_uuid: "c570c8606ab844a881ecda91c03cd6f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42939 } } peers { permanent_uuid: "1c84c0506b8c4e068fb33de4c37264ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34009 } }
I20260620 23:26:25.387328 19641 raft_consensus.cc:374] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:25.387425 19641 raft_consensus.cc:729] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c84c0506b8c4e068fb33de4c37264ab, State: Initialized, Role: FOLLOWER
I20260620 23:26:25.388070 19641 consensus_queue.cc:260] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35d135641f340528bbec4b287b3c29b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43867 } } peers { permanent_uuid: "c570c8606ab844a881ecda91c03cd6f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42939 } } peers { permanent_uuid: "1c84c0506b8c4e068fb33de4c37264ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34009 } }
I20260620 23:26:25.389227 19623 heartbeater.cc:503] Master 127.0.97.62:36041 was elected leader, sending a full tablet report...
I20260620 23:26:25.389138 19642 raft_consensus.cc:348] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35d135641f340528bbec4b287b3c29b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43867 } } peers { permanent_uuid: "c570c8606ab844a881ecda91c03cd6f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42939 } } peers { permanent_uuid: "1c84c0506b8c4e068fb33de4c37264ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34009 } }
I20260620 23:26:25.389554 19642 raft_consensus.cc:374] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:25.389684 19642 raft_consensus.cc:729] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c570c8606ab844a881ecda91c03cd6f2, State: Initialized, Role: FOLLOWER
I20260620 23:26:25.390081 19641 ts_tablet_manager.cc:1428] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260620 23:26:25.390403 19642 consensus_queue.cc:260] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35d135641f340528bbec4b287b3c29b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43867 } } peers { permanent_uuid: "c570c8606ab844a881ecda91c03cd6f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42939 } } peers { permanent_uuid: "1c84c0506b8c4e068fb33de4c37264ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34009 } }
I20260620 23:26:25.390411 19643 raft_consensus.cc:348] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35d135641f340528bbec4b287b3c29b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43867 } } peers { permanent_uuid: "c570c8606ab844a881ecda91c03cd6f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42939 } } peers { permanent_uuid: "1c84c0506b8c4e068fb33de4c37264ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34009 } }
I20260620 23:26:25.390838 19643 raft_consensus.cc:374] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:25.390942 19643 raft_consensus.cc:729] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35d135641f340528bbec4b287b3c29b, State: Initialized, Role: FOLLOWER
I20260620 23:26:25.391705 19643 consensus_queue.cc:260] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35d135641f340528bbec4b287b3c29b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43867 } } peers { permanent_uuid: "c570c8606ab844a881ecda91c03cd6f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42939 } } peers { permanent_uuid: "1c84c0506b8c4e068fb33de4c37264ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34009 } }
I20260620 23:26:25.394142 19642 ts_tablet_manager.cc:1428] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260620 23:26:25.394996 19493 heartbeater.cc:503] Master 127.0.97.62:36041 was elected leader, sending a full tablet report...
I20260620 23:26:25.395617 19643 ts_tablet_manager.cc:1428] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260620 23:26:25.469144 19649 raft_consensus.cc:482] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:26:25.469426 19649 raft_consensus.cc:504] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35d135641f340528bbec4b287b3c29b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43867 } } peers { permanent_uuid: "c570c8606ab844a881ecda91c03cd6f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42939 } } peers { permanent_uuid: "1c84c0506b8c4e068fb33de4c37264ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34009 } }
I20260620 23:26:25.471144 19649 leader_election.cc:288] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c570c8606ab844a881ecda91c03cd6f2 (127.0.97.1:42939), 1c84c0506b8c4e068fb33de4c37264ab (127.0.97.3:34009)
I20260620 23:26:25.479748 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.471573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37653 (local address 127.0.97.3:34009)
0620 23:26:25.471812 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:25.471820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:25.472400 (+ 580us) server_negotiation.cc:407] Connection header received
0620 23:26:25.472594 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:25.472597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:25.472659 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:25.472737 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:25.473249 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.474214 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.477308 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.478531 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.478662 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.478834 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:25.479352 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 23:26:25.479488 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:26:25.479753 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.471664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48627 (local address 127.0.97.1:42939)
0620 23:26:25.471910 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:25.471916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:25.472446 (+ 530us) server_negotiation.cc:407] Connection header received
0620 23:26:25.472915 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:25.472919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:25.472986 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:25.473055 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:25.473655 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.474591 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.477319 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.478525 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.478662 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.478806 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:25.479353 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 23:26:25.479494 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:26:25.480028 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.471494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34009 (local address 127.0.97.2:37653)
0620 23:26:25.472257 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.472286 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:25.472476 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:25.472776 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:25.472780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:25.472800 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:25.473106 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:25.473116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.474331 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.474335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:25.477010 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:25.477019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.478817 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.478825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:25.479113 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.479207 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 23:26:25.479309 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:26:25.479501 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":158,"spinlock_wait_cycles":22272,"thread_start_us":486,"threads_started":1}
I20260620 23:26:25.480064 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.471513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42939 (local address 127.0.97.2:48627)
0620 23:26:25.472280 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.472295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:25.472450 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:25.473107 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:25.473111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:25.473133 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:25.473497 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:25.473507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.474735 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.474743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:25.476996 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:25.477015 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.478838 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.478848 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:25.479110 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.479208 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 23:26:25.479309 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:26:25.479501 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":144,"thread_start_us":372,"threads_started":1}
I20260620 23:26:25.481081 19578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6854619733d4eae8dfb740fd4b4ad5c" candidate_uuid: "d35d135641f340528bbec4b287b3c29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c84c0506b8c4e068fb33de4c37264ab" is_pre_election: true
I20260620 23:26:25.481158 19317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6854619733d4eae8dfb740fd4b4ad5c" candidate_uuid: "d35d135641f340528bbec4b287b3c29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c570c8606ab844a881ecda91c03cd6f2" is_pre_election: true
I20260620 23:26:25.481642 19578 raft_consensus.cc:2454] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35d135641f340528bbec4b287b3c29b in term 0.
I20260620 23:26:25.481679 19317 raft_consensus.cc:2454] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35d135641f340528bbec4b287b3c29b in term 0.
I20260620 23:26:25.482345 19381 leader_election.cc:302] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c570c8606ab844a881ecda91c03cd6f2, d35d135641f340528bbec4b287b3c29b; no voters:
I20260620 23:26:25.482810 19649 raft_consensus.cc:2781] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:26:25.482960 19649 raft_consensus.cc:482] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:26:25.483070 19649 raft_consensus.cc:3037] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:25.485504 19649 raft_consensus.cc:504] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35d135641f340528bbec4b287b3c29b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43867 } } peers { permanent_uuid: "c570c8606ab844a881ecda91c03cd6f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42939 } } peers { permanent_uuid: "1c84c0506b8c4e068fb33de4c37264ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34009 } }
I20260620 23:26:25.486184 19649 leader_election.cc:288] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [CANDIDATE]: Term 1 election: Requested vote from peers c570c8606ab844a881ecda91c03cd6f2 (127.0.97.1:42939), 1c84c0506b8c4e068fb33de4c37264ab (127.0.97.3:34009)
I20260620 23:26:25.486665 19317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6854619733d4eae8dfb740fd4b4ad5c" candidate_uuid: "d35d135641f340528bbec4b287b3c29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c570c8606ab844a881ecda91c03cd6f2"
I20260620 23:26:25.486838 19578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6854619733d4eae8dfb740fd4b4ad5c" candidate_uuid: "d35d135641f340528bbec4b287b3c29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c84c0506b8c4e068fb33de4c37264ab"
I20260620 23:26:25.486939 19317 raft_consensus.cc:3037] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:25.487097 19578 raft_consensus.cc:3037] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:25.489267 19317 raft_consensus.cc:2454] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35d135641f340528bbec4b287b3c29b in term 1.
I20260620 23:26:25.489388 19578 raft_consensus.cc:2454] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35d135641f340528bbec4b287b3c29b in term 1.
I20260620 23:26:25.489712 19381 leader_election.cc:302] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c570c8606ab844a881ecda91c03cd6f2, d35d135641f340528bbec4b287b3c29b; no voters:
I20260620 23:26:25.490018 19649 raft_consensus.cc:2781] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:25.490511 19649 raft_consensus.cc:686] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 1 LEADER]: Becoming Leader. State: Replica: d35d135641f340528bbec4b287b3c29b, State: Running, Role: LEADER
I20260620 23:26:25.491043 19649 consensus_queue.cc:237] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35d135641f340528bbec4b287b3c29b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43867 } } peers { permanent_uuid: "c570c8606ab844a881ecda91c03cd6f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42939 } } peers { permanent_uuid: "1c84c0506b8c4e068fb33de4c37264ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34009 } }
I20260620 23:26:25.495824 19175 catalog_manager.cc:5495] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b reported cstate change: term changed from 0 to 1, leader changed from <none> to d35d135641f340528bbec4b287b3c29b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d35d135641f340528bbec4b287b3c29b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35d135641f340528bbec4b287b3c29b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c570c8606ab844a881ecda91c03cd6f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c84c0506b8c4e068fb33de4c37264ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34009 } health_report { overall_health: UNKNOWN } } }
W20260620 23:26:25.532650 19624 tablet.cc:2369] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:26:25.546164 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.540406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.0.97.2:43867)
0620 23:26:25.540653 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:25.540659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:25.540765 (+ 106us) server_negotiation.cc:407] Connection header received
0620 23:26:25.541016 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:25.541020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:25.541115 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:25.541293 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:26:25.541686 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.542414 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.543931 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.544366 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.544508 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.545072 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:26:25.545565 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:26:25.545757 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:25.545868 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 23:26:25.545913 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 23:26:25.554399 19578 raft_consensus.cc:1261] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab [term 1 FOLLOWER]: Refusing update from remote peer d35d135641f340528bbec4b287b3c29b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:26:25.554610 19317 raft_consensus.cc:1261] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2 [term 1 FOLLOWER]: Refusing update from remote peer d35d135641f340528bbec4b287b3c29b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:26:25.555541 19652 consensus_queue.cc:1035] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c84c0506b8c4e068fb33de4c37264ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:26:25.555860 19649 consensus_queue.cc:1035] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c570c8606ab844a881ecda91c03cd6f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:26:25.564127 19494 tablet.cc:2369] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:26:25.569257 19657 mvcc.cc:205] Tried to move back new op lower bound from 7299063748821434368 to 7299063748575367168. Current Snapshot: MvccSnapshot[applied={T|T < 7299063748821434368}]
I20260620 23:26:25.585062 19175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:26:25.585404 19175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:26:25.589262 19175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6f8a3a8683545aab09892d110aeb51f: Sending DeleteTablet for 3 replicas of tablet c6854619733d4eae8dfb740fd4b4ad5c
I20260620 23:26:25.590809 19558 tablet_service.cc:1514] Processing DeleteTablet for tablet c6854619733d4eae8dfb740fd4b4ad5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:25 UTC) from {username='slave'} at 127.0.0.1:40326
I20260620 23:26:25.590845 19297 tablet_service.cc:1514] Processing DeleteTablet for tablet c6854619733d4eae8dfb740fd4b4ad5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:25 UTC) from {username='slave'} at 127.0.0.1:57162
I20260620 23:26:25.591012 19428 tablet_service.cc:1514] Processing DeleteTablet for tablet c6854619733d4eae8dfb740fd4b4ad5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:25 UTC) from {username='slave'} at 127.0.0.1:38672
I20260620 23:26:25.593324 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 19235
I20260620 23:26:25.608286 19664 tablet_replica.cc:330] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2: stopping tablet replica
I20260620 23:26:25.608816 19664 raft_consensus.cc:2229] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:26:25.609408 19664 raft_consensus.cc:2258] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:25.610074 19666 tablet_replica.cc:330] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b: stopping tablet replica
I20260620 23:26:25.610203 19665 tablet_replica.cc:330] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab: stopping tablet replica
I20260620 23:26:25.610646 19665 raft_consensus.cc:2229] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:26:25.610656 19666 raft_consensus.cc:2229] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:26:25.611166 19665 raft_consensus.cc:2258] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:25.611490 19666 raft_consensus.cc:2258] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:25.612326 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.597338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.0.97.1:42939)
0620 23:26:25.597583 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:25.597591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:25.599377 (+ 1786us) server_negotiation.cc:407] Connection header received
0620 23:26:25.600270 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:25.600278 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:25.600416 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:25.600510 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:25.601947 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.603624 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.607759 (+ 4135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.608489 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.608630 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.610514 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:25.610526 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:25.610544 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:25.611370 (+ 826us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:25.611435 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:25.611461 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:25.611550 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:25.611813 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 23:26:25.611946 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:26:25.613583 19357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:25.614004 19665 ts_tablet_manager.cc:1905] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:25.618055 19666 ts_tablet_manager.cc:1905] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:25.622835 19665 ts_tablet_manager.cc:1918] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:25.623028 19665 log.cc:1196] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/c6854619733d4eae8dfb740fd4b4ad5c
I20260620 23:26:25.623565 19665 ts_tablet_manager.cc:1939] T c6854619733d4eae8dfb740fd4b4ad5c P 1c84c0506b8c4e068fb33de4c37264ab: Deleting consensus metadata
I20260620 23:26:25.625588 19163 catalog_manager.cc:4841] TS 1c84c0506b8c4e068fb33de4c37264ab (127.0.97.3:34009): tablet c6854619733d4eae8dfb740fd4b4ad5c (table test-table [id=0fd23e11c402475492e70493c5592cf3]) successfully deleted
I20260620 23:26:25.625820 19666 ts_tablet_manager.cc:1918] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:25.626003 19666 log.cc:1196] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/c6854619733d4eae8dfb740fd4b4ad5c
I20260620 23:26:25.626494 19666 ts_tablet_manager.cc:1939] T c6854619733d4eae8dfb740fd4b4ad5c P d35d135641f340528bbec4b287b3c29b: Deleting consensus metadata
I20260620 23:26:25.628053 19161 catalog_manager.cc:4841] TS d35d135641f340528bbec4b287b3c29b (127.0.97.2:43867): tablet c6854619733d4eae8dfb740fd4b4ad5c (table test-table [id=0fd23e11c402475492e70493c5592cf3]) successfully deleted
I20260620 23:26:25.744199 19664 ts_tablet_manager.cc:1905] T c6854619733d4eae8dfb740fd4b4ad5c P c570c8606ab844a881ecda91c03cd6f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:25.748041 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 19235
I20260620 23:26:25.771497 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 19366
W20260620 23:26:25.771539 19160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42939 (error 108)
I20260620 23:26:25.774085 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.773550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42939 (local address 127.0.0.1:57194)
0620 23:26:25.773751 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.773878 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 23:26:25.774386 19160 catalog_manager.cc:4584] TS c570c8606ab844a881ecda91c03cd6f2 (127.0.97.1:42939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6854619733d4eae8dfb740fd4b4ad5c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42939: connect: Connection refused (error 111)
I20260620 23:26:25.780368 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.772336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.0.97.2:43867)
0620 23:26:25.772546 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:25.772552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:25.773013 (+ 461us) server_negotiation.cc:407] Connection header received
0620 23:26:25.773103 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:25.773109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:25.773246 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:25.773376 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:25.774000 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.775158 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.776971 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.777424 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.777535 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.778877 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:25.778887 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:25.778901 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:25.779550 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:25.779606 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:25.779635 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:25.779726 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:25.779883 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 23:26:25.779964 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:26:25.781121 19488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:25.828114 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.827607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42939 (local address 127.0.0.1:57204)
0620 23:26:25.827848 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.827948 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 23:26:25.829144 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.828823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42939 (local address 127.0.0.1:57206)
0620 23:26:25.828962 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.829029 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 23:26:25.889987 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.889483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42939 (local address 127.0.0.1:57208)
0620 23:26:25.889710 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.889817 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 23:26:25.891148 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.890808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42939 (local address 127.0.0.1:57216)
0620 23:26:25.890968 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.891034 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:26:25.896876 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 19366
I20260620 23:26:25.916321 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 19496
I20260620 23:26:25.923583 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.917181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.0.97.3:34009)
0620 23:26:25.917356 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:25.917360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:25.917379 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:26:25.917587 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:25.917591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:25.917706 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:25.917801 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:25.918246 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.919036 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.920520 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:25.920954 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:25.921059 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:25.922178 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:25.922190 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:25.922212 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:25.922943 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:25.922993 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:25.923020 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:25.923107 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:25.923233 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 23:26:25.923323 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:26:25.924214 19618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:25.971472 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.970978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42939 (local address 127.0.0.1:57218)
0620 23:26:25.971186 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.971288 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 23:26:25.972548 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:25.972162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42939 (local address 127.0.0.1:57226)
0620 23:26:25.972358 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:25.972430 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 23:26:26.015077 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 19496
I20260620 23:26:26.033923 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 19145
I20260620 23:26:26.035148 19206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:26.147495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 19145
I20260620 23:26:26.147642 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:26.147115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42939 (local address 127.0.0.1:57232)
0620 23:26:26.147311 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:26.147432 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
2026-06-20T23:26:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2970 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:26 dist-test-slave-5kzz krb5kdc[19675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:26 dist-test-slave-5kzz krb5kdc[19675](info): set up 2 sockets
Jun 20 23:26:26 dist-test-slave-5kzz krb5kdc[19675](info): commencing operation
krb5kdc: starting...
W20260620 23:26:27.267081 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:27 dist-test-slave-5kzz krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:26:27.526170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41663
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:27.872011 19688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:27.872345 19688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:27.872510 19688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:27.879936 19688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:27.880057 19688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:27.880149 19688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:27.880239 19688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:27.880306 19688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:27.890354 19688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:27.891949 19688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:27.893419 19688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:27.905772 19696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:27.905833 19693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:27.905853 19694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:27.906898 19688 server_base.cc:1034] running on GCE node
I20260620 23:26:27.907784 19688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:27.909276 19688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:27.910547 19688 hybrid_clock.cc:648] HybridClock initialized: now 1781997987910478 us; error 55 us; skew 500 ppm
I20260620 23:26:27.910993 19688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:27 dist-test-slave-5kzz krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:27.914317 19688 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:26:27.915861 19688 webserver.cc:460] Webserver started at http://127.0.97.62:42859/ using document root <none> and password file <none>
I20260620 23:26:27.916522 19688 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:27.916620 19688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:27.916867 19688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:27.920014 19688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "62202a0e388f4fbb9ed5e5d9e39eb195"
format_stamp: "Formatted at 2026-06-20 23:26:27 on dist-test-slave-5kzz"
server_key: "89d9665095f441e925afc24122989301"
server_key_iv: "60ac536051b7fbb0ebde0991111b7a55"
server_key_version: "encryptionkey@0"
I20260620 23:26:27.920747 19688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "62202a0e388f4fbb9ed5e5d9e39eb195"
format_stamp: "Formatted at 2026-06-20 23:26:27 on dist-test-slave-5kzz"
server_key: "89d9665095f441e925afc24122989301"
server_key_iv: "60ac536051b7fbb0ebde0991111b7a55"
server_key_version: "encryptionkey@0"
I20260620 23:26:27.926034 19688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:26:27.929770 19703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:27.931391 19688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:26:27.931578 19688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "62202a0e388f4fbb9ed5e5d9e39eb195"
format_stamp: "Formatted at 2026-06-20 23:26:27 on dist-test-slave-5kzz"
server_key: "89d9665095f441e925afc24122989301"
server_key_iv: "60ac536051b7fbb0ebde0991111b7a55"
server_key_version: "encryptionkey@0"
I20260620 23:26:27.931799 19688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:27.979617 19688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:27.984334 19688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:27.984651 19688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:28.004402 19688 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41663
I20260620 23:26:28.004432 19754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41663 every 8 connection(s)
I20260620 23:26:28.005846 19688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:28.009779 19755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:28.011914 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 19688
I20260620 23:26:28.012127 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:28.012562 388 external_mini_cluster.cc:1428] Setting key a3f34c7abfde6bc30f85e86b08b2b92b
I20260620 23:26:28.023073 19755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195: Bootstrap starting.
I20260620 23:26:28.027204 19755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195: Neither blocks nor log segments found. Creating new log.
Jun 20 23:26:28 dist-test-slave-5kzz krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:28.028682 19755 log.cc:824] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:28.032460 19755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195: No bootstrap required, opened a new log
I20260620 23:26:28.038590 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:28.014411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.0.97.62:41663)
0620 23:26:28.015364 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:28.015376 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:28.015418 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:26:28.017116 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:28.017138 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:28.017531 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:28.017846 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:28.019893 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:28.021824 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:28.023789 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:28.024535 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:28.024719 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:28.029854 (+ 5135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:28.029862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:28.029881 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:28.029921 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:28.034033 (+ 4112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:28.034862 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:28.034873 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:28.034886 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:28.035115 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:28.035634 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:28.035643 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:28.035649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:28.036281 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:28.036671 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:28.037113 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 23:26:28.037567 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":511,"threads_started":1}
I20260620 23:26:28.041888 19755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62202a0e388f4fbb9ed5e5d9e39eb195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41663 } }
I20260620 23:26:28.042268 19755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:28.042387 19755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62202a0e388f4fbb9ed5e5d9e39eb195, State: Initialized, Role: FOLLOWER
I20260620 23:26:28.042999 19755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62202a0e388f4fbb9ed5e5d9e39eb195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41663 } }
I20260620 23:26:28.043232 19755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:28.043367 19755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:28.043532 19755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:28.045679 19755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62202a0e388f4fbb9ed5e5d9e39eb195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41663 } }
I20260620 23:26:28.046191 19755 leader_election.cc:302] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62202a0e388f4fbb9ed5e5d9e39eb195; no voters:
I20260620 23:26:28.046967 19755 leader_election.cc:288] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:28.047082 19760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:28.047768 19760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [term 1 LEADER]: Becoming Leader. State: Replica: 62202a0e388f4fbb9ed5e5d9e39eb195, State: Running, Role: LEADER
I20260620 23:26:28.048326 19760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62202a0e388f4fbb9ed5e5d9e39eb195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41663 } }
I20260620 23:26:28.049269 19755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:28.051496 19762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62202a0e388f4fbb9ed5e5d9e39eb195. Latest consensus state: current_term: 1 leader_uuid: "62202a0e388f4fbb9ed5e5d9e39eb195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62202a0e388f4fbb9ed5e5d9e39eb195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41663 } } }
I20260620 23:26:28.052029 19762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:28.051397 19761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62202a0e388f4fbb9ed5e5d9e39eb195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62202a0e388f4fbb9ed5e5d9e39eb195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41663 } } }
I20260620 23:26:28.059767 19761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:28.060235 19767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:28.067126 19767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:28.076866 19767 catalog_manager.cc:1347] Generated new cluster ID: 63054e5f38504d2f8dd1ebfc336c2cae
I20260620 23:26:28.077005 19767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:28.096549 19767 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:28.097939 19767 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:28.110210 19767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195: Generated new TSK 0
I20260620 23:26:28.111469 19767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:26:28.171025 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41663
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:28.513329 19781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:28.513644 19781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:28.513772 19781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:28.521490 19781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:28.521626 19781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:28.521821 19781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:26:28.531843 19781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:28.533257 19781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:28.534632 19781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:28.548921 19787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:28.549109 19789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:28.549044 19786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:28.550182 19781 server_base.cc:1034] running on GCE node
I20260620 23:26:28.550935 19781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:28.551944 19781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:28.553224 19781 hybrid_clock.cc:648] HybridClock initialized: now 1781997988553172 us; error 45 us; skew 500 ppm
I20260620 23:26:28.553567 19781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:28 dist-test-slave-5kzz krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:28.556986 19781 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:26:28.558652 19781 webserver.cc:460] Webserver started at http://127.0.97.1:33159/ using document root <none> and password file <none>
I20260620 23:26:28.559442 19781 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:28.559612 19781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:28.560024 19781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:28.563560 19781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "ca3322ea11404462af4964efc87aee3a"
format_stamp: "Formatted at 2026-06-20 23:26:28 on dist-test-slave-5kzz"
server_key: "3f23e02728badf5deb9cb7fc1b006c59"
server_key_iv: "f47beacf1b6e6d1a8a8c0cc6d0e47c1c"
server_key_version: "encryptionkey@0"
I20260620 23:26:28.564391 19781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ca3322ea11404462af4964efc87aee3a"
format_stamp: "Formatted at 2026-06-20 23:26:28 on dist-test-slave-5kzz"
server_key: "3f23e02728badf5deb9cb7fc1b006c59"
server_key_iv: "f47beacf1b6e6d1a8a8c0cc6d0e47c1c"
server_key_version: "encryptionkey@0"
I20260620 23:26:28.569873 19781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:26:28.573781 19796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:28.575387 19781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:26:28.575584 19781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "ca3322ea11404462af4964efc87aee3a"
format_stamp: "Formatted at 2026-06-20 23:26:28 on dist-test-slave-5kzz"
server_key: "3f23e02728badf5deb9cb7fc1b006c59"
server_key_iv: "f47beacf1b6e6d1a8a8c0cc6d0e47c1c"
server_key_version: "encryptionkey@0"
I20260620 23:26:28.575806 19781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:28.595871 19781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:28.600502 19781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:28.600852 19781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:28.601912 19781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:28.603384 19781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:28.603488 19781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:28.603649 19781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:28.603724 19781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:28.639144 19781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44115
I20260620 23:26:28.639240 19908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44115 every 8 connection(s)
I20260620 23:26:28.640511 19781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:26:28.646479 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 19781
I20260620 23:26:28.646664 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:26:28.647235 388 external_mini_cluster.cc:1428] Setting key 1509ca0d0290f577c1b69dd6312a4673
Jun 20 23:26:28 dist-test-slave-5kzz krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:28.666923 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:28.643480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52079 (local address 127.0.97.62:41663)
0620 23:26:28.643974 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:28.643979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:28.645103 (+ 1124us) server_negotiation.cc:407] Connection header received
0620 23:26:28.647601 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:28.647605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:28.647680 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:28.647772 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:28.649735 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:28.650609 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:28.653294 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:28.653789 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:28.653950 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:28.659067 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:28.659075 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:28.659083 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:28.659136 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:28.663165 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:28.664116 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:28.664126 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:28.664132 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:28.664297 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:28.664751 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:28.664758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:28.664764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:28.665151 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:28.665327 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:28.666112 (+ 785us) server_negotiation.cc:299] Negotiation successful
0620 23:26:28.666387 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":257,"threads_started":1}
I20260620 23:26:28.667701 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:28.644096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41663 (local address 127.0.97.1:52079)
0620 23:26:28.644837 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:28.644876 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:28.647195 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:28.648075 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:28.648088 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:28.648721 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:28.649534 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:28.649560 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:28.650805 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:28.650813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:28.653111 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:28.653129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:28.654068 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:28.654076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:28.654422 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:28.655407 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:26:28.655451 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:28.658740 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:28.663387 (+ 4647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:28.663396 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:28.663412 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:28.663943 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:28.664422 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:28.664426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:28.664428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:28.664628 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:28.665310 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:28.665317 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:28.665689 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 23:26:28.665966 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 23:26:28.666414 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":302,"threads_started":1}
I20260620 23:26:28.669796 19909 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41663
I20260620 23:26:28.670141 19909 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:28.671022 19909 heartbeater.cc:511] Master 127.0.97.62:41663 requested a full tablet report, sending...
I20260620 23:26:28.673806 19720 ts_manager.cc:194] Registered new tserver with Master: ca3322ea11404462af4964efc87aee3a (127.0.97.1:44115)
I20260620 23:26:28.676146 19720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52079
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:26:28.711572 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41663
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:29.045002 19915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:29.045298 19915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:29.045475 19915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:29.053035 19915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:29.053157 19915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:29.053339 19915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:26:29.063876 19915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:29.065225 19915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:29.066541 19915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:29.080145 19920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:29.080777 19921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:29.080835 19923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:29.080924 19915 server_base.cc:1034] running on GCE node
I20260620 23:26:29.081627 19915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:29.082592 19915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:29.083803 19915 hybrid_clock.cc:648] HybridClock initialized: now 1781997989083728 us; error 67 us; skew 500 ppm
I20260620 23:26:29.084244 19915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:29 dist-test-slave-5kzz krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:29.087646 19915 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:26:29.089282 19915 webserver.cc:460] Webserver started at http://127.0.97.2:38599/ using document root <none> and password file <none>
I20260620 23:26:29.090042 19915 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:29.090149 19915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:29.090410 19915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:29.093492 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "fc1a7a0825374a1e87898a7101a0db7b"
format_stamp: "Formatted at 2026-06-20 23:26:29 on dist-test-slave-5kzz"
server_key: "4c0744477cff8b66e225ebb102b383a4"
server_key_iv: "152000a77050f41868539db300592829"
server_key_version: "encryptionkey@0"
I20260620 23:26:29.094252 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fc1a7a0825374a1e87898a7101a0db7b"
format_stamp: "Formatted at 2026-06-20 23:26:29 on dist-test-slave-5kzz"
server_key: "4c0744477cff8b66e225ebb102b383a4"
server_key_iv: "152000a77050f41868539db300592829"
server_key_version: "encryptionkey@0"
I20260620 23:26:29.099727 19915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:26:29.103510 19930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:29.105010 19915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:26:29.105211 19915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "fc1a7a0825374a1e87898a7101a0db7b"
format_stamp: "Formatted at 2026-06-20 23:26:29 on dist-test-slave-5kzz"
server_key: "4c0744477cff8b66e225ebb102b383a4"
server_key_iv: "152000a77050f41868539db300592829"
server_key_version: "encryptionkey@0"
I20260620 23:26:29.105445 19915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:29.123215 19915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:29.127848 19915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:29.128165 19915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:29.129196 19915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:29.130657 19915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:29.130801 19915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:29.130955 19915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:29.131028 19915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:29.165156 19915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34321
I20260620 23:26:29.165195 20042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34321 every 8 connection(s)
I20260620 23:26:29.166499 19915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:26:29.167188 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 19915
I20260620 23:26:29.167383 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:26:29.167973 388 external_mini_cluster.cc:1428] Setting key 662d6e6d56d5a14cc80fc19b2899a98e
Jun 20 23:26:29 dist-test-slave-5kzz krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:29.187829 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.169700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42979 (local address 127.0.97.62:41663)
0620 23:26:29.170170 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:29.170177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:29.171359 (+ 1182us) server_negotiation.cc:407] Connection header received
0620 23:26:29.173255 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:29.173261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:29.173355 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:29.173478 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:29.175058 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.175815 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.177387 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.177819 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.177923 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.181888 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:29.181894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:29.181897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:29.181932 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:29.184639 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:29.185420 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:29.185427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:29.185435 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:29.185557 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:29.186009 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:29.186019 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:29.186021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:29.186279 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:29.186436 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:29.187202 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 23:26:29.187447 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":247,"threads_started":1}
I20260620 23:26:29.188843 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.170328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41663 (local address 127.0.97.2:42979)
0620 23:26:29.171200 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:29.171245 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:29.172964 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:29.173719 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:29.173727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:29.174246 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:29.174887 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.174908 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.175987 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.175991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.177227 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.177239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.178061 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.178071 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.178422 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.179293 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:26:29.179321 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:29.181628 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:29.184793 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:29.184802 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:29.184820 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:29.185286 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:29.185690 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:29.185695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:29.185699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:29.185884 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:29.186411 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:29.186419 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:29.186842 (+ 423us) client_negotiation.cc:769] Sending connection context
0620 23:26:29.187122 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 23:26:29.187667 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":419,"threads_started":1}
I20260620 23:26:29.190557 20043 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41663
I20260620 23:26:29.190936 20043 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:29.191725 20043 heartbeater.cc:511] Master 127.0.97.62:41663 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:26:29.193329 19720 ts_manager.cc:194] Registered new tserver with Master: fc1a7a0825374a1e87898a7101a0db7b (127.0.97.2:34321)
I20260620 23:26:29.194280 19720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:26:29.227677 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41663
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:29.568194 20049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:29.568547 20049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:29.568724 20049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:29.577304 20049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:29.577553 20049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:29.577782 20049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:26:29.589455 20049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:29.590926 20049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:29.592275 20049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:29.606284 20054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:29.606223 20055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:29.606310 20057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:29.607293 20049 server_base.cc:1034] running on GCE node
I20260620 23:26:29.607908 20049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:29.608979 20049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:29.610255 20049 hybrid_clock.cc:648] HybridClock initialized: now 1781997989610214 us; error 84 us; skew 500 ppm
I20260620 23:26:29.610622 20049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:29 dist-test-slave-5kzz krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:29.614120 20049 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:26:29.615656 20049 webserver.cc:460] Webserver started at http://127.0.97.3:43721/ using document root <none> and password file <none>
I20260620 23:26:29.616367 20049 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:29.616496 20049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:29.616825 20049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:29.619808 20049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "9560c152601047e6931170e6964f9145"
format_stamp: "Formatted at 2026-06-20 23:26:29 on dist-test-slave-5kzz"
server_key: "8db2ee69867f5b1de0e2ebfbd6044ca3"
server_key_iv: "4d82df4a8e373e9c72bcbfe34893d1f9"
server_key_version: "encryptionkey@0"
I20260620 23:26:29.620586 20049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9560c152601047e6931170e6964f9145"
format_stamp: "Formatted at 2026-06-20 23:26:29 on dist-test-slave-5kzz"
server_key: "8db2ee69867f5b1de0e2ebfbd6044ca3"
server_key_iv: "4d82df4a8e373e9c72bcbfe34893d1f9"
server_key_version: "encryptionkey@0"
I20260620 23:26:29.625906 20049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:26:29.629530 20064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:29.631085 20049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:26:29.631318 20049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "9560c152601047e6931170e6964f9145"
format_stamp: "Formatted at 2026-06-20 23:26:29 on dist-test-slave-5kzz"
server_key: "8db2ee69867f5b1de0e2ebfbd6044ca3"
server_key_iv: "4d82df4a8e373e9c72bcbfe34893d1f9"
server_key_version: "encryptionkey@0"
I20260620 23:26:29.631511 20049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:29.647509 20049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:29.653223 20049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:29.653553 20049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:29.654999 20049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:29.656670 20049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:29.656754 20049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:29.656884 20049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:29.656958 20049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:29.679857 19909 heartbeater.cc:503] Master 127.0.97.62:41663 was elected leader, sending a full tablet report...
I20260620 23:26:29.693846 20049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37385
I20260620 23:26:29.693889 20176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37385 every 8 connection(s)
I20260620 23:26:29.695245 20049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:26:29.703961 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 20049
I20260620 23:26:29.704157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:26:29.704592 388 external_mini_cluster.cc:1428] Setting key a798c443ac557137cac8c1d1fc2e6689
Jun 20 23:26:29 dist-test-slave-5kzz krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:29.715802 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.697912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47285 (local address 127.0.97.62:41663)
0620 23:26:29.698359 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:29.698363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:29.699427 (+ 1064us) server_negotiation.cc:407] Connection header received
0620 23:26:29.701241 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:29.701246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:29.701328 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:29.701432 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:29.703172 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.703984 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.705786 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.706225 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.706361 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.710089 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:29.710094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:29.710097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:29.710136 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:29.712963 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:29.713688 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:29.713692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:29.713694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:29.713815 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:29.714201 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:29.714205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:29.714207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:29.714464 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:29.714576 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:29.715346 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 23:26:29.715521 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":226,"threads_started":1}
I20260620 23:26:29.716679 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.698455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41663 (local address 127.0.97.3:47285)
0620 23:26:29.699255 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:29.699294 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:29.700970 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:29.701724 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:29.701733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:29.702275 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:29.702975 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.702997 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.704162 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.704166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.705638 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.705651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.706457 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.706465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.706781 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.707479 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:26:29.707502 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:29.709847 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:29.713108 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:29.713116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:29.713129 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:29.713572 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:29.713906 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:26:29.713910 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:26:29.713912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:26:29.714090 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:26:29.714566 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:29.714572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:29.714975 (+ 403us) client_negotiation.cc:769] Sending connection context
0620 23:26:29.715214 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 23:26:29.715593 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":352,"threads_started":1}
I20260620 23:26:29.718226 20177 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41663
I20260620 23:26:29.718493 20177 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:29.719385 20177 heartbeater.cc:511] Master 127.0.97.62:41663 requested a full tablet report, sending...
I20260620 23:26:29.720831 19720 ts_manager.cc:194] Registered new tserver with Master: 9560c152601047e6931170e6964f9145 (127.0.97.3:37385)
I20260620 23:26:29.721647 19720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47285
I20260620 23:26:29.732486 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:26:29.749579 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.736869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60874 (local address 127.0.97.62:41663)
0620 23:26:29.737103 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:29.737112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:29.737494 (+ 382us) server_negotiation.cc:407] Connection header received
0620 23:26:29.737699 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:29.737704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:29.737781 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:29.737885 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:29.739411 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.740157 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.741613 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.742014 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.742110 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.744623 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:29.744627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:29.744631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:29.744663 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:29.747141 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:29.747859 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:29.747863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:29.747865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:29.747980 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:29.748308 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:29.748311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:29.748313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:29.748614 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:29.748787 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:29.749081 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 23:26:29.749262 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:26:29.753435 19720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:26:29.760457 19720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:26:29.786224 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.774032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.0.97.2:34321)
0620 23:26:29.774721 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:29.774735 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:29.777582 (+ 2847us) server_negotiation.cc:407] Connection header received
0620 23:26:29.777996 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:29.778009 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:29.778315 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:29.778559 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:29.779302 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.780946 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.783520 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.785015 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.785157 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.785379 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:29.785713 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 23:26:29.785894 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":986,"threads_started":1}
I20260620 23:26:29.786271 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.772670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34321 (local address 127.0.0.1:55748)
0620 23:26:29.777425 (+ 4755us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:29.777450 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:29.777746 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:29.778610 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:29.778618 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:29.778648 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:29.779076 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.779092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.781084 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.781089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.783114 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.783127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.785219 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.785225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.785423 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.785479 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 23:26:29.785621 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 23:26:29.785819 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4587,"mutex_wait_us":38,"thread_start_us":2784,"threads_started":1}
I20260620 23:26:29.787150 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.772476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44115 (local address 127.0.0.1:41192)
0620 23:26:29.773509 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:29.773558 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:29.773821 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:29.779133 (+ 5312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:29.779138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:29.779178 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:29.779700 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.779723 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.781689 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.781698 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.784392 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.784405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.786282 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.786289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.786507 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.786536 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:26:29.786598 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:26:29.786779 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":124,"thread_start_us":5248,"threads_started":1}
I20260620 23:26:29.787760 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.774753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60862 (local address 127.0.97.3:37385)
0620 23:26:29.775609 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:29.775621 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:29.778150 (+ 2529us) server_negotiation.cc:407] Connection header received
0620 23:26:29.778538 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:29.778546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:29.778814 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:29.779012 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:29.779945 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.781597 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.784791 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.786173 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.786317 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.786547 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:29.787077 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 23:26:29.787231 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":509,"threads_started":1}
I20260620 23:26:29.788190 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.773530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37385 (local address 127.0.0.1:60862)
0620 23:26:29.778006 (+ 4476us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:29.778027 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:29.778277 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:29.779006 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:29.779010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:29.779036 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:29.779694 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.779709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.781745 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.781754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.784639 (+ 2885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.784646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.786726 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.786734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.786882 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.786908 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:26:29.787021 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:26:29.787162 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4346,"mutex_wait_us":117,"thread_start_us":3820,"threads_started":1}
I20260620 23:26:29.788201 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.772680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.0.97.1:44115)
0620 23:26:29.778443 (+ 5763us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:29.778454 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:29.778481 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:26:29.778591 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:29.778600 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:29.778886 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:29.779946 (+ 1060us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:29.780225 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.781530 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.784572 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.786023 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.787412 (+ 1389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.787632 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:29.787718 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 23:26:29.787891 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5583,"thread_start_us":5385,"threads_started":1}
I20260620 23:26:29.790597 19978 tablet_service.cc:1467] Processing CreateTablet for tablet ead666899b1548679d0010555cd9f9fe (DEFAULT_TABLE table=test-table [id=49fe5f1f17fe493ca765cc39f965b9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:29.792205 19978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead666899b1548679d0010555cd9f9fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:29.791868 20112 tablet_service.cc:1467] Processing CreateTablet for tablet ead666899b1548679d0010555cd9f9fe (DEFAULT_TABLE table=test-table [id=49fe5f1f17fe493ca765cc39f965b9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:29.793304 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead666899b1548679d0010555cd9f9fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:29.794018 19844 tablet_service.cc:1467] Processing CreateTablet for tablet ead666899b1548679d0010555cd9f9fe (DEFAULT_TABLE table=test-table [id=49fe5f1f17fe493ca765cc39f965b9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:29.795356 19844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead666899b1548679d0010555cd9f9fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:29.806515 20196 tablet_bootstrap.cc:492] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a: Bootstrap starting.
I20260620 23:26:29.811162 20196 tablet_bootstrap.cc:654] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:29.811631 20198 tablet_bootstrap.cc:492] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145: Bootstrap starting.
I20260620 23:26:29.812866 20196 log.cc:824] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:29.813478 20197 tablet_bootstrap.cc:492] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b: Bootstrap starting.
I20260620 23:26:29.815306 20198 tablet_bootstrap.cc:654] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:29.816387 20196 tablet_bootstrap.cc:492] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a: No bootstrap required, opened a new log
I20260620 23:26:29.816623 20198 log.cc:824] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:29.816685 20196 ts_tablet_manager.cc:1397] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 23:26:29.817358 20197 tablet_bootstrap.cc:654] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:29.818574 20197 log.cc:824] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:29.821177 20197 tablet_bootstrap.cc:492] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b: No bootstrap required, opened a new log
I20260620 23:26:29.821453 20197 ts_tablet_manager.cc:1397] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260620 23:26:29.825495 20198 tablet_bootstrap.cc:492] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145: No bootstrap required, opened a new log
I20260620 23:26:29.825781 20198 ts_tablet_manager.cc:1397] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260620 23:26:29.829061 20197 raft_consensus.cc:348] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3322ea11404462af4964efc87aee3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44115 } } peers { permanent_uuid: "fc1a7a0825374a1e87898a7101a0db7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34321 } } peers { permanent_uuid: "9560c152601047e6931170e6964f9145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37385 } }
I20260620 23:26:29.829576 20197 raft_consensus.cc:374] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:29.829321 20196 raft_consensus.cc:348] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3322ea11404462af4964efc87aee3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44115 } } peers { permanent_uuid: "fc1a7a0825374a1e87898a7101a0db7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34321 } } peers { permanent_uuid: "9560c152601047e6931170e6964f9145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37385 } }
I20260620 23:26:29.829679 20197 raft_consensus.cc:729] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1a7a0825374a1e87898a7101a0db7b, State: Initialized, Role: FOLLOWER
I20260620 23:26:29.829818 20196 raft_consensus.cc:374] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:29.829919 20196 raft_consensus.cc:729] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3322ea11404462af4964efc87aee3a, State: Initialized, Role: FOLLOWER
I20260620 23:26:29.830489 20197 consensus_queue.cc:260] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3322ea11404462af4964efc87aee3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44115 } } peers { permanent_uuid: "fc1a7a0825374a1e87898a7101a0db7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34321 } } peers { permanent_uuid: "9560c152601047e6931170e6964f9145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37385 } }
I20260620 23:26:29.830513 20196 consensus_queue.cc:260] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3322ea11404462af4964efc87aee3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44115 } } peers { permanent_uuid: "fc1a7a0825374a1e87898a7101a0db7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34321 } } peers { permanent_uuid: "9560c152601047e6931170e6964f9145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37385 } }
I20260620 23:26:29.831601 20043 heartbeater.cc:503] Master 127.0.97.62:41663 was elected leader, sending a full tablet report...
I20260620 23:26:29.833495 20197 ts_tablet_manager.cc:1428] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260620 23:26:29.835134 20196 ts_tablet_manager.cc:1428] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20260620 23:26:29.837368 20198 raft_consensus.cc:348] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3322ea11404462af4964efc87aee3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44115 } } peers { permanent_uuid: "fc1a7a0825374a1e87898a7101a0db7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34321 } } peers { permanent_uuid: "9560c152601047e6931170e6964f9145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37385 } }
I20260620 23:26:29.837736 20198 raft_consensus.cc:374] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:29.837870 20198 raft_consensus.cc:729] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9560c152601047e6931170e6964f9145, State: Initialized, Role: FOLLOWER
I20260620 23:26:29.838500 20198 consensus_queue.cc:260] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3322ea11404462af4964efc87aee3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44115 } } peers { permanent_uuid: "fc1a7a0825374a1e87898a7101a0db7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34321 } } peers { permanent_uuid: "9560c152601047e6931170e6964f9145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37385 } }
I20260620 23:26:29.839512 20177 heartbeater.cc:503] Master 127.0.97.62:41663 was elected leader, sending a full tablet report...
I20260620 23:26:29.839916 20198 ts_tablet_manager.cc:1428] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145: Time spent starting tablet: real 0.014s user 0.002s sys 0.014s
W20260620 23:26:29.893057 19910 tablet.cc:2369] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:26:29.919199 20044 tablet.cc:2369] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:26:29.931056 20202 raft_consensus.cc:482] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:26:29.931439 20202 raft_consensus.cc:504] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3322ea11404462af4964efc87aee3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44115 } } peers { permanent_uuid: "fc1a7a0825374a1e87898a7101a0db7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34321 } } peers { permanent_uuid: "9560c152601047e6931170e6964f9145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37385 } }
I20260620 23:26:29.933313 20202 leader_election.cc:288] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca3322ea11404462af4964efc87aee3a (127.0.97.1:44115), 9560c152601047e6931170e6964f9145 (127.0.97.3:37385)
I20260620 23:26:29.940732 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.933780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52069 (local address 127.0.97.1:44115)
0620 23:26:29.934044 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:29.934050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:29.934877 (+ 827us) server_negotiation.cc:407] Connection header received
0620 23:26:29.935158 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:29.935165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:29.935248 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:29.935364 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:29.936068 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.936978 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.938941 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.939721 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.939880 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.939991 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:29.940316 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:26:29.940461 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 23:26:29.940753 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.933863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38437 (local address 127.0.97.3:37385)
0620 23:26:29.934093 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:29.934101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:29.934949 (+ 848us) server_negotiation.cc:407] Connection header received
0620 23:26:29.935208 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:29.935217 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:29.935277 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:29.935370 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:29.936216 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.937024 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.938926 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.939701 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.939854 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.939968 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:29.940326 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 23:26:29.940460 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:26:29.940925 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.933706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44115 (local address 127.0.97.2:52069)
0620 23:26:29.934767 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:29.934784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:29.934981 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:29.935424 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:29.935431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:29.935455 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:29.935917 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.935931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.937106 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.937112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.938792 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.938804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.939955 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.939960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.940147 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.940183 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:26:29.940289 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:26:29.940463 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":37,"thread_start_us":648,"threads_started":1}
I20260620 23:26:29.940948 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.933753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37385 (local address 127.0.97.2:38437)
0620 23:26:29.934825 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:29.934844 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:29.935058 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:29.935395 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:29.935399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:29.935424 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:29.935984 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.935995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.937143 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.937150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.938766 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:29.938781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.939935 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.939941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:29.940151 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.940196 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 23:26:29.940298 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:26:29.940460 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":63,"thread_start_us":498,"threads_started":1}
I20260620 23:26:29.942232 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead666899b1548679d0010555cd9f9fe" candidate_uuid: "fc1a7a0825374a1e87898a7101a0db7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9560c152601047e6931170e6964f9145" is_pre_election: true
I20260620 23:26:29.942233 19864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead666899b1548679d0010555cd9f9fe" candidate_uuid: "fc1a7a0825374a1e87898a7101a0db7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3322ea11404462af4964efc87aee3a" is_pre_election: true
I20260620 23:26:29.942858 19864 raft_consensus.cc:2454] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc1a7a0825374a1e87898a7101a0db7b in term 0.
I20260620 23:26:29.942858 20132 raft_consensus.cc:2454] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc1a7a0825374a1e87898a7101a0db7b in term 0.
I20260620 23:26:29.943565 19931 leader_election.cc:302] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca3322ea11404462af4964efc87aee3a, fc1a7a0825374a1e87898a7101a0db7b; no voters:
I20260620 23:26:29.943984 20202 raft_consensus.cc:2781] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:26:29.944139 20202 raft_consensus.cc:482] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:26:29.944264 20202 raft_consensus.cc:3037] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:29.946410 20202 raft_consensus.cc:504] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3322ea11404462af4964efc87aee3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44115 } } peers { permanent_uuid: "fc1a7a0825374a1e87898a7101a0db7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34321 } } peers { permanent_uuid: "9560c152601047e6931170e6964f9145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37385 } }
W20260620 23:26:29.946895 20178 tablet.cc:2369] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:26:29.947136 20202 leader_election.cc:288] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [CANDIDATE]: Term 1 election: Requested vote from peers ca3322ea11404462af4964efc87aee3a (127.0.97.1:44115), 9560c152601047e6931170e6964f9145 (127.0.97.3:37385)
I20260620 23:26:29.947587 19864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead666899b1548679d0010555cd9f9fe" candidate_uuid: "fc1a7a0825374a1e87898a7101a0db7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3322ea11404462af4964efc87aee3a"
I20260620 23:26:29.947836 19864 raft_consensus.cc:3037] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:29.948016 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead666899b1548679d0010555cd9f9fe" candidate_uuid: "fc1a7a0825374a1e87898a7101a0db7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9560c152601047e6931170e6964f9145"
I20260620 23:26:29.948235 20132 raft_consensus.cc:3037] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:29.951308 19864 raft_consensus.cc:2454] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc1a7a0825374a1e87898a7101a0db7b in term 1.
I20260620 23:26:29.951308 20132 raft_consensus.cc:2454] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc1a7a0825374a1e87898a7101a0db7b in term 1.
I20260620 23:26:29.951761 19931 leader_election.cc:302] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca3322ea11404462af4964efc87aee3a, fc1a7a0825374a1e87898a7101a0db7b; no voters:
I20260620 23:26:29.952075 20202 raft_consensus.cc:2781] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:29.952509 20202 raft_consensus.cc:686] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 1 LEADER]: Becoming Leader. State: Replica: fc1a7a0825374a1e87898a7101a0db7b, State: Running, Role: LEADER
I20260620 23:26:29.952998 20202 consensus_queue.cc:237] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3322ea11404462af4964efc87aee3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44115 } } peers { permanent_uuid: "fc1a7a0825374a1e87898a7101a0db7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34321 } } peers { permanent_uuid: "9560c152601047e6931170e6964f9145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37385 } }
I20260620 23:26:29.957494 19719 catalog_manager.cc:5495] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b reported cstate change: term changed from 0 to 1, leader changed from <none> to fc1a7a0825374a1e87898a7101a0db7b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "fc1a7a0825374a1e87898a7101a0db7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3322ea11404462af4964efc87aee3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc1a7a0825374a1e87898a7101a0db7b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9560c152601047e6931170e6964f9145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37385 } health_report { overall_health: UNKNOWN } } }
I20260620 23:26:29.981508 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:29.976113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.0.97.2:34321)
0620 23:26:29.976357 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:29.976364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:29.976384 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:26:29.976478 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:29.976482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:29.976556 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:29.976722 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:26:29.977102 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.977820 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.979261 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:29.979710 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:29.979841 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:29.980380 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:26:29.980899 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:26:29.981063 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:29.981119 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 23:26:29.981199 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:26:29.989635 20132 raft_consensus.cc:1261] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145 [term 1 FOLLOWER]: Refusing update from remote peer fc1a7a0825374a1e87898a7101a0db7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:26:29.989645 19864 raft_consensus.cc:1261] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a [term 1 FOLLOWER]: Refusing update from remote peer fc1a7a0825374a1e87898a7101a0db7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:26:29.990636 20207 consensus_queue.cc:1035] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3322ea11404462af4964efc87aee3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:26:29.991048 20202 consensus_queue.cc:1035] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9560c152601047e6931170e6964f9145" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:26:30.000463 20210 mvcc.cc:205] Tried to move back new op lower bound from 7299063766987624448 to 7299063766851215360. Current Snapshot: MvccSnapshot[applied={T|T < 7299063766987624448}]
I20260620 23:26:30.004215 20212 mvcc.cc:205] Tried to move back new op lower bound from 7299063766987624448 to 7299063766851215360. Current Snapshot: MvccSnapshot[applied={T|T < 7299063766987624448}]
I20260620 23:26:30.019596 19720 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:26:30.019989 19720 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:26:30.022787 20211 mvcc.cc:205] Tried to move back new op lower bound from 7299063766987624448 to 7299063766851215360. Current Snapshot: MvccSnapshot[applied={T|T < 7299063766987624448}]
I20260620 23:26:30.025365 19720 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62202a0e388f4fbb9ed5e5d9e39eb195: Sending DeleteTablet for 3 replicas of tablet ead666899b1548679d0010555cd9f9fe
I20260620 23:26:30.026576 19844 tablet_service.cc:1514] Processing DeleteTablet for tablet ead666899b1548679d0010555cd9f9fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41192
I20260620 23:26:30.028666 19978 tablet_service.cc:1514] Processing DeleteTablet for tablet ead666899b1548679d0010555cd9f9fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55748
I20260620 23:26:30.032585 20221 tablet_replica.cc:330] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a: stopping tablet replica
I20260620 23:26:30.033107 19977 tablet_service.cc:1514] Processing DeleteTablet for tablet ead666899b1548679d0010555cd9f9fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55748
I20260620 23:26:30.034128 20112 tablet_service.cc:1514] Processing DeleteTablet for tablet ead666899b1548679d0010555cd9f9fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60862
I20260620 23:26:30.034860 20222 tablet_replica.cc:330] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b: stopping tablet replica
I20260620 23:26:30.036566 20223 tablet_replica.cc:330] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145: stopping tablet replica
I20260620 23:26:30.035903 20222 raft_consensus.cc:2229] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:26:30.037020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 19781
I20260620 23:26:30.037707 20222 raft_consensus.cc:2258] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:30.038321 20223 raft_consensus.cc:2229] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:26:30.039665 20223 raft_consensus.cc:2258] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:30.042165 20222 ts_tablet_manager.cc:1905] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:30.045658 20223 ts_tablet_manager.cc:1905] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:30.046671 20221 raft_consensus.cc:2229] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a [term 1 FOLLOWER]: Raft consensus shutting down.
Jun 20 23:26:30 dist-test-slave-5kzz krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:26:30.050352 20221 raft_consensus.cc:2258] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:30.054227 20223 ts_tablet_manager.cc:1918] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:30.054405 20223 log.cc:1196] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/ead666899b1548679d0010555cd9f9fe
I20260620 23:26:30.054991 20223 ts_tablet_manager.cc:1939] T ead666899b1548679d0010555cd9f9fe P 9560c152601047e6931170e6964f9145: Deleting consensus metadata
I20260620 23:26:30.055244 20221 ts_tablet_manager.cc:1905] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:30.056927 19706 catalog_manager.cc:4841] TS 9560c152601047e6931170e6964f9145 (127.0.97.3:37385): tablet ead666899b1548679d0010555cd9f9fe (table test-table [id=49fe5f1f17fe493ca765cc39f965b9a6]) successfully deleted
I20260620 23:26:30.058918 20222 ts_tablet_manager.cc:1918] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:30.059171 20222 log.cc:1196] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/ead666899b1548679d0010555cd9f9fe
I20260620 23:26:30.059810 20222 ts_tablet_manager.cc:1939] T ead666899b1548679d0010555cd9f9fe P fc1a7a0825374a1e87898a7101a0db7b: Deleting consensus metadata
I20260620 23:26:30.062389 19705 catalog_manager.cc:4841] TS fc1a7a0825374a1e87898a7101a0db7b (127.0.97.2:34321): tablet ead666899b1548679d0010555cd9f9fe (table test-table [id=49fe5f1f17fe493ca765cc39f965b9a6]) successfully deleted
W20260620 23:26:30.063050 19705 catalog_manager.cc:4808] TS fc1a7a0825374a1e87898a7101a0db7b (127.0.97.2:34321): delete failed for tablet ead666899b1548679d0010555cd9f9fe because the tablet was not found. No further retry: Not found: Tablet not found: ead666899b1548679d0010555cd9f9fe
I20260620 23:26:30.063405 20221 ts_tablet_manager.cc:1918] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:30.063597 20221 log.cc:1196] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997680534866-388-0/minicluster-data/ts-0/wal/wals/ead666899b1548679d0010555cd9f9fe
I20260620 23:26:30.064126 20221 ts_tablet_manager.cc:1939] T ead666899b1548679d0010555cd9f9fe P ca3322ea11404462af4964efc87aee3a: Deleting consensus metadata
I20260620 23:26:30.066020 19704 catalog_manager.cc:4841] TS ca3322ea11404462af4964efc87aee3a (127.0.97.1:44115): tablet ead666899b1548679d0010555cd9f9fe (table test-table [id=49fe5f1f17fe493ca765cc39f965b9a6]) successfully deleted
I20260620 23:26:30.067242 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:30.038081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.0.97.1:44115)
0620 23:26:30.038252 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:30.038258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:30.039372 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 23:26:30.039647 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:30.039654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:30.039768 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:30.039854 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:30.041677 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:30.042845 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:30.044708 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:30.045385 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:30.045483 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:30.058510 (+ 13027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:30.058530 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:30.058551 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:30.058605 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:30.063646 (+ 5041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:30.064543 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:30.064553 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:30.064557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:30.064712 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:30.065122 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:30.065128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:30.065131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:30.065508 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:30.065832 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:30.066632 (+ 800us) server_negotiation.cc:299] Negotiation successful
0620 23:26:30.066887 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:26:30.067930 19904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:30.160176 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 19781
I20260620 23:26:30.183506 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 19915
Jun 20 23:26:30 dist-test-slave-5kzz krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:26:30.204532 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:30.184333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.0.97.2:34321)
0620 23:26:30.184525 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:30.184531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:30.184552 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:26:30.184623 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:30.184628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:30.184737 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:30.184850 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:30.186420 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:30.187795 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:30.189804 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:30.190532 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:30.190851 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:30.196697 (+ 5846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:30.196710 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:30.196730 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:30.196787 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:30.201425 (+ 4638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:30.202396 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:30.202405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:30.202412 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:30.202595 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:30.203059 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:30.203063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:30.203065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:30.203420 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:30.203687 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:30.204022 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 23:26:30.204200 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:26:30.205379 20038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:30.322248 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 19915
I20260620 23:26:30.341279 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 20049
Jun 20 23:26:30 dist-test-slave-5kzz krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:26:30.356801 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:30.342050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.0.97.3:37385)
0620 23:26:30.342251 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:30.342255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:30.342274 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:26:30.342427 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:30.342431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:30.342497 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:30.342608 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:30.343929 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:30.344693 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:30.346016 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:30.346400 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:30.346495 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:30.351219 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:30.351235 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:30.351251 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:30.351290 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:30.354046 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:30.354826 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:30.354835 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:30.354837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:30.354995 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:30.355378 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:30.355382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:30.355385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:30.355769 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:30.355992 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:30.356367 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 23:26:30.356540 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:26:30.357457 20172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:30.448081 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 20049
I20260620 23:26:30.465029 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 19688
I20260620 23:26:30.466233 19750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:30.577767 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 19688
2026-06-20T23:26:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4430 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T23:26:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:30Z Disabled control of system clock
I20260620 23:26:30.655748 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38533
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:26:30.995003 20232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:30.995324 20232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:30.995478 20232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:31.003333 20232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:31.003463 20232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:31.003568 20232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:31.003669 20232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:31.003724 20232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:31.013607 20232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:31.015084 20232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:31.017421 20232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:31.030866 20237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:31.030929 20238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:31.030872 20240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:31.031400 20232 server_base.cc:1034] running on GCE node
I20260620 23:26:31.032721 20232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:31.034332 20232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:31.035602 20232 hybrid_clock.cc:648] HybridClock initialized: now 1781997991035530 us; error 65 us; skew 500 ppm
I20260620 23:26:31.036000 20232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:26:31.038358 20232 webserver.cc:460] Webserver started at http://127.0.97.62:32961/ using document root <none> and password file <none>
I20260620 23:26:31.039188 20232 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:31.039330 20232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:31.039683 20232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:31.043087 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "e657b257e40d4a8bb4f0c620bf9b35d0"
format_stamp: "Formatted at 2026-06-20 23:26:31 on dist-test-slave-5kzz"
server_key: "190478c6755d491bbf9ed9efe078e564"
server_key_iv: "3664b81b536bd8cd4b3a76d8be563baa"
server_key_version: "encryptionkey@0"
I20260620 23:26:31.043996 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "e657b257e40d4a8bb4f0c620bf9b35d0"
format_stamp: "Formatted at 2026-06-20 23:26:31 on dist-test-slave-5kzz"
server_key: "190478c6755d491bbf9ed9efe078e564"
server_key_iv: "3664b81b536bd8cd4b3a76d8be563baa"
server_key_version: "encryptionkey@0"
I20260620 23:26:31.049942 20232 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 23:26:31.056201 20246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:31.058112 20232 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:26:31.058360 20232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "e657b257e40d4a8bb4f0c620bf9b35d0"
format_stamp: "Formatted at 2026-06-20 23:26:31 on dist-test-slave-5kzz"
server_key: "190478c6755d491bbf9ed9efe078e564"
server_key_iv: "3664b81b536bd8cd4b3a76d8be563baa"
server_key_version: "encryptionkey@0"
I20260620 23:26:31.058822 20232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:31.078419 20232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:31.079622 20232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:31.079950 20232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:31.104465 20232 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38533
I20260620 23:26:31.104485 20297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38533 every 8 connection(s)
I20260620 23:26:31.105931 20232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:31.109916 20298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:31.111565 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 20232
I20260620 23:26:31.111786 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:31.112257 388 external_mini_cluster.cc:1428] Setting key 332e52ec5f77633195b4f3c5ca52cf4e
I20260620 23:26:31.125909 20298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0: Bootstrap starting.
I20260620 23:26:31.128413 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:31.115056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.0.97.62:38533)
0620 23:26:31.116761 (+ 1705us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:31.116774 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:31.116826 (+ 52us) server_negotiation.cc:407] Connection header received
0620 23:26:31.118618 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:31.118646 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:31.119343 (+ 697us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:31.119817 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:31.120448 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:31.121854 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:31.123469 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:31.124251 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:31.124477 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:31.125814 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:31.125821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:31.125838 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:31.126535 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:31.126621 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:31.126645 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:31.126838 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:31.127063 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 23:26:31.127418 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1168,"thread_start_us":617,"threads_started":1}
I20260620 23:26:31.129649 20298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:31.130853 20298 log.cc:824] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:31.134119 20298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0: No bootstrap required, opened a new log
I20260620 23:26:31.142349 20298 raft_consensus.cc:348] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e657b257e40d4a8bb4f0c620bf9b35d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38533 } }
I20260620 23:26:31.142930 20298 raft_consensus.cc:374] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:31.143095 20298 raft_consensus.cc:729] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e657b257e40d4a8bb4f0c620bf9b35d0, State: Initialized, Role: FOLLOWER
I20260620 23:26:31.143821 20298 consensus_queue.cc:260] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e657b257e40d4a8bb4f0c620bf9b35d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38533 } }
I20260620 23:26:31.144093 20298 raft_consensus.cc:388] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:31.144245 20298 raft_consensus.cc:482] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:31.144445 20298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:31.146835 20298 raft_consensus.cc:504] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e657b257e40d4a8bb4f0c620bf9b35d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38533 } }
I20260620 23:26:31.147401 20298 leader_election.cc:302] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e657b257e40d4a8bb4f0c620bf9b35d0; no voters:
I20260620 23:26:31.148161 20298 leader_election.cc:288] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:31.148378 20303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:31.149281 20303 raft_consensus.cc:686] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [term 1 LEADER]: Becoming Leader. State: Replica: e657b257e40d4a8bb4f0c620bf9b35d0, State: Running, Role: LEADER
I20260620 23:26:31.149991 20303 consensus_queue.cc:237] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e657b257e40d4a8bb4f0c620bf9b35d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38533 } }
I20260620 23:26:31.150831 20298 sys_catalog.cc:564] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:31.153465 20304 sys_catalog.cc:455] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e657b257e40d4a8bb4f0c620bf9b35d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e657b257e40d4a8bb4f0c620bf9b35d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38533 } } }
I20260620 23:26:31.153972 20304 sys_catalog.cc:458] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:31.153519 20305 sys_catalog.cc:455] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e657b257e40d4a8bb4f0c620bf9b35d0. Latest consensus state: current_term: 1 leader_uuid: "e657b257e40d4a8bb4f0c620bf9b35d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e657b257e40d4a8bb4f0c620bf9b35d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38533 } } }
I20260620 23:26:31.154148 20305 sys_catalog.cc:458] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:31.156570 20309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:31.162557 20309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:31.175125 20309 catalog_manager.cc:1347] Generated new cluster ID: 5eb55f79999e4ef19c24683405cb8a76
I20260620 23:26:31.175380 20309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:31.193498 20309 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:31.194989 20309 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:31.209021 20309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0: Generated new TSK 0
I20260620 23:26:31.210007 20309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:31.221876 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38533
--builtin_ntp_servers=127.0.97.20:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:26:31.586611 20322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:31.586990 20322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:31.587141 20322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:31.594661 20322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:31.594839 20322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:31.595009 20322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:26:31.604846 20322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:31.606218 20322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:31.608055 20322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:31.621922 20330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:31.621894 20328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:31.621985 20327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:31.622812 20322 server_base.cc:1034] running on GCE node
I20260620 23:26:31.623498 20322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:31.624507 20322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:31.625757 20322 hybrid_clock.cc:648] HybridClock initialized: now 1781997991625709 us; error 32 us; skew 500 ppm
I20260620 23:26:31.626111 20322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:26:31.632370 20322 webserver.cc:460] Webserver started at http://127.0.97.1:40329/ using document root <none> and password file <none>
I20260620 23:26:31.633167 20322 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:31.633306 20322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:31.633638 20322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:31.636646 20322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/data/instance:
uuid: "cfa2be09cc29404baf9be9c207b59d7c"
format_stamp: "Formatted at 2026-06-20 23:26:31 on dist-test-slave-5kzz"
server_key: "4436b25ebf94ea88a70a6585176048a0"
server_key_iv: "770d01be37713e383088ea36cb870944"
server_key_version: "encryptionkey@0"
I20260620 23:26:31.637472 20322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cfa2be09cc29404baf9be9c207b59d7c"
format_stamp: "Formatted at 2026-06-20 23:26:31 on dist-test-slave-5kzz"
server_key: "4436b25ebf94ea88a70a6585176048a0"
server_key_iv: "770d01be37713e383088ea36cb870944"
server_key_version: "encryptionkey@0"
I20260620 23:26:31.642938 20322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260620 23:26:31.646417 20336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:31.647845 20322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:26:31.648028 20322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/wal
uuid: "cfa2be09cc29404baf9be9c207b59d7c"
format_stamp: "Formatted at 2026-06-20 23:26:31 on dist-test-slave-5kzz"
server_key: "4436b25ebf94ea88a70a6585176048a0"
server_key_iv: "770d01be37713e383088ea36cb870944"
server_key_version: "encryptionkey@0"
I20260620 23:26:31.648260 20322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:31.677249 20322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:31.678177 20322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:31.678463 20322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:31.679513 20322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:31.681049 20322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:31.681178 20322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:31.681313 20322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:31.681397 20322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:31.716962 20322 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46157
I20260620 23:26:31.717060 20448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46157 every 8 connection(s)
I20260620 23:26:31.718268 20322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:26:31.723342 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 20322
I20260620 23:26:31.723547 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:26:31.724023 388 external_mini_cluster.cc:1428] Setting key 6e1c987495bec0a28d204faf3d4a628a
I20260620 23:26:31.726778 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38533
--builtin_ntp_servers=127.0.97.20:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:26:31.731431 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:31.721035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56109 (local address 127.0.97.62:38533)
0620 23:26:31.721510 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:31.721514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:31.722728 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 23:26:31.724764 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:31.724770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:31.724864 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:31.724985 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:31.726067 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:31.726833 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:31.728460 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:31.728881 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:31.729018 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:31.730498 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:31.730503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:31.730507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:31.730527 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:31.730567 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:31.730578 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:31.730683 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:31.730948 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 23:26:31.731051 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":268,"threads_started":1}
I20260620 23:26:31.732506 20452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:31.721589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38533 (local address 127.0.97.1:56109)
0620 23:26:31.722538 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:31.722573 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:31.724474 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:31.725231 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:31.725239 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:31.725333 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:31.725903 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:31.725919 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:31.726985 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:31.726992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:31.728297 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:31.728312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:31.729128 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:31.729135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:31.729377 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:31.730136 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:26:31.730139 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:31.730190 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:26:31.730275 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:31.730711 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:31.730725 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:31.730736 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:26:31.730937 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 23:26:31.731368 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":330,"threads_started":1}
I20260620 23:26:31.734139 20449 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38533
I20260620 23:26:31.734469 20449 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:31.735296 20449 heartbeater.cc:511] Master 127.0.97.62:38533 requested a full tablet report, sending...
I20260620 23:26:31.737483 20263 ts_manager.cc:194] Registered new tserver with Master: cfa2be09cc29404baf9be9c207b59d7c (127.0.97.1:46157)
I20260620 23:26:31.739147 20263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:56109
W20260620 23:26:32.079819 20453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:32.080184 20453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:32.080344 20453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:32.089937 20453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:32.090193 20453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:32.090415 20453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:26:32.100931 20453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:32.102346 20453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:32.104400 20453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:32.119707 20459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:32.119598 20458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:32.122435 20461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:32.125190 20453 server_base.cc:1034] running on GCE node
I20260620 23:26:32.126151 20453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:32.127316 20453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:32.128624 20453 hybrid_clock.cc:648] HybridClock initialized: now 1781997992128570 us; error 61 us; skew 500 ppm
I20260620 23:26:32.129349 20453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:26:32.132319 20453 webserver.cc:460] Webserver started at http://127.0.97.2:40447/ using document root <none> and password file <none>
I20260620 23:26:32.133105 20453 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:32.133246 20453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:32.133672 20453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:32.137066 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/data/instance:
uuid: "f5a89a1b1e8748ff863449bc37109b85"
format_stamp: "Formatted at 2026-06-20 23:26:32 on dist-test-slave-5kzz"
server_key: "9cecfcaa07afc53c4b78caad002ae410"
server_key_iv: "1c0d694a25eb27ac6f73442a58f39c34"
server_key_version: "encryptionkey@0"
I20260620 23:26:32.138075 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f5a89a1b1e8748ff863449bc37109b85"
format_stamp: "Formatted at 2026-06-20 23:26:32 on dist-test-slave-5kzz"
server_key: "9cecfcaa07afc53c4b78caad002ae410"
server_key_iv: "1c0d694a25eb27ac6f73442a58f39c34"
server_key_version: "encryptionkey@0"
I20260620 23:26:32.144134 20453 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 23:26:32.148712 20467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:32.150599 20453 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:26:32.150864 20453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/wal
uuid: "f5a89a1b1e8748ff863449bc37109b85"
format_stamp: "Formatted at 2026-06-20 23:26:32 on dist-test-slave-5kzz"
server_key: "9cecfcaa07afc53c4b78caad002ae410"
server_key_iv: "1c0d694a25eb27ac6f73442a58f39c34"
server_key_version: "encryptionkey@0"
I20260620 23:26:32.151136 20453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:32.179382 20453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:32.180286 20453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:32.180603 20453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:32.181720 20453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:32.184020 20453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:32.184235 20453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:32.184393 20453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:32.184516 20453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:32.223744 20453 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46741
I20260620 23:26:32.223847 20579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46741 every 8 connection(s)
I20260620 23:26:32.225059 20453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:26:32.225515 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 20453
I20260620 23:26:32.225700 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:26:32.226346 388 external_mini_cluster.cc:1428] Setting key b6c6d6802d85ef166152e0872a00ce3a
I20260620 23:26:32.230140 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38533
--builtin_ntp_servers=127.0.97.20:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:26:32.240759 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.228339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39215 (local address 127.0.97.62:38533)
0620 23:26:32.228748 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:32.228755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:32.230098 (+ 1343us) server_negotiation.cc:407] Connection header received
0620 23:26:32.233248 (+ 3150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:32.233253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:32.233337 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:32.233485 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:32.234861 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.235639 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.237426 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.237870 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.238052 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.239855 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:32.239860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:32.239864 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:32.239885 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:32.239917 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:32.239930 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:32.240041 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:32.240288 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 23:26:32.240404 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260620 23:26:32.241916 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.228682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38533 (local address 127.0.97.2:39215)
0620 23:26:32.229931 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:32.229988 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:32.232833 (+ 2845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:32.233799 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:32.233810 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:32.233927 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:32.234602 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.234623 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.235823 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.235830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.237248 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.237263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.238155 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.238163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.238442 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.239467 (+ 1025us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:26:32.239470 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:32.239521 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:26:32.239614 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:32.240040 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:32.240053 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:32.240064 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:26:32.240263 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 23:26:32.240713 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":320,"threads_started":1}
I20260620 23:26:32.243767 20580 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38533
I20260620 23:26:32.244138 20580 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:32.244952 20580 heartbeater.cc:511] Master 127.0.97.62:38533 requested a full tablet report, sending...
I20260620 23:26:32.246562 20263 ts_manager.cc:194] Registered new tserver with Master: f5a89a1b1e8748ff863449bc37109b85 (127.0.97.2:46741)
I20260620 23:26:32.247457 20263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:39215
W20260620 23:26:32.581622 20583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:32.581962 20583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:32.582104 20583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:32.589644 20583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:32.589792 20583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:32.589973 20583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:26:32.599570 20583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:32.600915 20583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:32.602759 20583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:32.616043 20591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:32.616117 20588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:32.616139 20589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:32.616935 20583 server_base.cc:1034] running on GCE node
I20260620 23:26:32.617632 20583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:32.618670 20583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:32.619936 20583 hybrid_clock.cc:648] HybridClock initialized: now 1781997992619904 us; error 48 us; skew 500 ppm
I20260620 23:26:32.620262 20583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:26:32.622552 20583 webserver.cc:460] Webserver started at http://127.0.97.3:41067/ using document root <none> and password file <none>
I20260620 23:26:32.623524 20583 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:32.623625 20583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:32.624012 20583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:32.627041 20583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/data/instance:
uuid: "40a34844dfb246caad6b4c66c247f0b4"
format_stamp: "Formatted at 2026-06-20 23:26:32 on dist-test-slave-5kzz"
server_key: "9c975595984abd028e1fdcc868171d55"
server_key_iv: "a2446e3583d69aadc9cfe78e557aed07"
server_key_version: "encryptionkey@0"
I20260620 23:26:32.627842 20583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/wal/instance:
uuid: "40a34844dfb246caad6b4c66c247f0b4"
format_stamp: "Formatted at 2026-06-20 23:26:32 on dist-test-slave-5kzz"
server_key: "9c975595984abd028e1fdcc868171d55"
server_key_iv: "a2446e3583d69aadc9cfe78e557aed07"
server_key_version: "encryptionkey@0"
I20260620 23:26:32.632915 20583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:26:32.636416 20597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:32.637728 20583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:26:32.637912 20583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/wal
uuid: "40a34844dfb246caad6b4c66c247f0b4"
format_stamp: "Formatted at 2026-06-20 23:26:32 on dist-test-slave-5kzz"
server_key: "9c975595984abd028e1fdcc868171d55"
server_key_iv: "a2446e3583d69aadc9cfe78e557aed07"
server_key_version: "encryptionkey@0"
I20260620 23:26:32.638125 20583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:32.660688 20583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:32.661550 20583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:32.661845 20583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:32.663059 20583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:26:32.664709 20583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:26:32.664796 20583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:32.664958 20583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:26:32.665047 20583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:32.699855 20583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40149
I20260620 23:26:32.700021 20709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40149 every 8 connection(s)
I20260620 23:26:32.701215 20583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:26:32.709487 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 20583
I20260620 23:26:32.709667 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:26:32.710173 388 external_mini_cluster.cc:1428] Setting key b6bd7fbfb2609728a435f6e2423d377f
I20260620 23:26:32.714851 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.703861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50299 (local address 127.0.97.62:38533)
0620 23:26:32.704048 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:32.704052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:32.705483 (+ 1431us) server_negotiation.cc:407] Connection header received
0620 23:26:32.707347 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:32.707352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:32.707435 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:32.707533 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:32.708757 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.709846 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.711743 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.712294 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.712393 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.714032 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:32.714038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:32.714042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:32.714064 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:32.714107 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:32.714127 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:32.714206 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:32.714404 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 23:26:32.714530 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:26:32.715878 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.704372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38533 (local address 127.0.97.3:50299)
0620 23:26:32.705280 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:32.705316 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:32.707064 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:32.707812 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:32.707821 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:32.707922 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:26:32.708570 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.708589 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.710006 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.710010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.711599 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.711612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.712533 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.712543 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.712882 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.713624 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:26:32.713627 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:26:32.713668 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:26:32.713778 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:26:32.714219 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:26:32.714226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:26:32.714232 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:26:32.714376 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 23:26:32.714775 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":287,"threads_started":1}
I20260620 23:26:32.717398 20710 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38533
I20260620 23:26:32.717801 20710 heartbeater.cc:464] Registering TS with master...
I20260620 23:26:32.718942 20710 heartbeater.cc:511] Master 127.0.97.62:38533 requested a full tablet report, sending...
I20260620 23:26:32.720614 20263 ts_manager.cc:194] Registered new tserver with Master: 40a34844dfb246caad6b4c66c247f0b4 (127.0.97.3:40149)
I20260620 23:26:32.721470 20263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:50299
I20260620 23:26:32.725659 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:26:32.736646 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.730037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.0.97.62:38533)
0620 23:26:32.730298 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:32.730305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:32.730610 (+ 305us) server_negotiation.cc:407] Connection header received
0620 23:26:32.730883 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:32.730888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:32.730986 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:32.731099 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:32.731643 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.732604 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.734087 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.734527 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.734680 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.735869 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:32.735875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:32.735878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:32.735898 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:32.735931 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:32.735944 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:32.736054 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:32.736232 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 23:26:32.736329 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 23:26:32.742399 20449 heartbeater.cc:503] Master 127.0.97.62:38533 was elected leader, sending a full tablet report...
I20260620 23:26:32.741070 20263 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:26:32.748847 20263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:26:32.789047 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.764443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.0.97.1:46157)
0620 23:26:32.772901 (+ 8458us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:32.772916 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:32.780228 (+ 7312us) server_negotiation.cc:407] Connection header received
0620 23:26:32.780433 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:32.780441 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:32.780703 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:32.780896 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:32.781795 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.783552 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.786004 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.787412 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.787552 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.787844 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:32.788525 (+ 681us) server_negotiation.cc:299] Negotiation successful
0620 23:26:32.788705 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8170,"thread_start_us":8008,"threads_started":1}
I20260620 23:26:32.789086 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.763363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46157 (local address 127.0.0.1:39492)
0620 23:26:32.780072 (+ 16709us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:32.780092 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:32.780255 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:32.781180 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:32.781183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:32.781202 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:32.781617 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.781629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.783708 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.783713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.785799 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.785814 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.787998 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.788008 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.788214 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.788320 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 23:26:32.788456 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 23:26:32.788675 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16585,"mutex_wait_us":185,"thread_start_us":18369,"threads_started":1}
I20260620 23:26:32.789618 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.762949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40149 (local address 127.0.0.1:47554)
0620 23:26:32.778229 (+ 15280us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:32.778310 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:32.778559 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:32.780449 (+ 1890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:32.780457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:32.780486 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:32.780954 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.780970 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.783782 (+ 2812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.783789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.786216 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.786230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.788032 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.788039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.788246 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.788335 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 23:26:32.788450 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 23:26:32.788676 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14827,"mutex_wait_us":178,"thread_start_us":15880,"threads_started":1}
I20260620 23:26:32.789680 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.763275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47554 (local address 127.0.97.3:40149)
0620 23:26:32.779204 (+ 15929us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:32.779218 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:32.779272 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:26:32.779412 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:32.779424 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:32.779772 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:32.780030 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:32.782051 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.783615 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.786427 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.787821 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.787975 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.788229 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:32.789087 (+ 858us) server_negotiation.cc:299] Negotiation successful
0620 23:26:32.789269 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15682,"thread_start_us":18021,"threads_started":1}
I20260620 23:26:32.789994 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.763976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46741 (local address 127.0.0.1:41636)
0620 23:26:32.782037 (+ 18061us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:32.782055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:32.782197 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:32.782725 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:32.782729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:32.782755 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:32.783086 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.783095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.784816 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.784821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.787036 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.787049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.789198 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.789204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.789391 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.789424 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:26:32.789506 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 23:26:32.789631 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17944,"mutex_wait_us":1,"thread_start_us":15812,"threads_started":1}
I20260620 23:26:32.793090 20384 tablet_service.cc:1467] Processing CreateTablet for tablet fe2b3bbb03c8431f8f276130432a1084 (DEFAULT_TABLE table=test-table [id=56313ea655cb4b14a389e578575e3ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:32.793087 20645 tablet_service.cc:1467] Processing CreateTablet for tablet fe2b3bbb03c8431f8f276130432a1084 (DEFAULT_TABLE table=test-table [id=56313ea655cb4b14a389e578575e3ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:32.793906 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.765020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.0.97.2:46741)
0620 23:26:32.773417 (+ 8397us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:32.773429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:32.782181 (+ 8752us) server_negotiation.cc:407] Connection header received
0620 23:26:32.782340 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:32.782346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:32.782515 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:32.782733 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:32.783249 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.784666 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.787246 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.788682 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.788787 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.789012 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:32.790270 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0620 23:26:32.790438 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8200,"thread_start_us":8065,"threads_started":1}
I20260620 23:26:32.794480 20384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2b3bbb03c8431f8f276130432a1084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:32.795018 20645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2b3bbb03c8431f8f276130432a1084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:32.798135 20515 tablet_service.cc:1467] Processing CreateTablet for tablet fe2b3bbb03c8431f8f276130432a1084 (DEFAULT_TABLE table=test-table [id=56313ea655cb4b14a389e578575e3ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:26:32.799615 20515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2b3bbb03c8431f8f276130432a1084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:32.806492 20728 tablet_bootstrap.cc:492] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4: Bootstrap starting.
I20260620 23:26:32.809576 20729 tablet_bootstrap.cc:492] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c: Bootstrap starting.
I20260620 23:26:32.810253 20728 tablet_bootstrap.cc:654] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:32.811723 20728 log.cc:824] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:32.813793 20729 tablet_bootstrap.cc:654] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:32.815600 20729 log.cc:824] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:32.817834 20731 tablet_bootstrap.cc:492] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85: Bootstrap starting.
I20260620 23:26:32.819737 20729 tablet_bootstrap.cc:492] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c: No bootstrap required, opened a new log
I20260620 23:26:32.820042 20729 ts_tablet_manager.cc:1397] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20260620 23:26:32.820583 20728 tablet_bootstrap.cc:492] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4: No bootstrap required, opened a new log
I20260620 23:26:32.820886 20728 ts_tablet_manager.cc:1397] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260620 23:26:32.822762 20731 tablet_bootstrap.cc:654] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:32.824139 20731 log.cc:824] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:32.827680 20731 tablet_bootstrap.cc:492] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85: No bootstrap required, opened a new log
I20260620 23:26:32.828025 20731 ts_tablet_manager.cc:1397] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260620 23:26:32.829492 20728 raft_consensus.cc:348] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a34844dfb246caad6b4c66c247f0b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40149 } } peers { permanent_uuid: "cfa2be09cc29404baf9be9c207b59d7c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46157 } } peers { permanent_uuid: "f5a89a1b1e8748ff863449bc37109b85" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46741 } }
I20260620 23:26:32.829895 20728 raft_consensus.cc:374] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:32.830015 20728 raft_consensus.cc:729] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a34844dfb246caad6b4c66c247f0b4, State: Initialized, Role: FOLLOWER
I20260620 23:26:32.830549 20729 raft_consensus.cc:348] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a34844dfb246caad6b4c66c247f0b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40149 } } peers { permanent_uuid: "cfa2be09cc29404baf9be9c207b59d7c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46157 } } peers { permanent_uuid: "f5a89a1b1e8748ff863449bc37109b85" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46741 } }
I20260620 23:26:32.830765 20728 consensus_queue.cc:260] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a34844dfb246caad6b4c66c247f0b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40149 } } peers { permanent_uuid: "cfa2be09cc29404baf9be9c207b59d7c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46157 } } peers { permanent_uuid: "f5a89a1b1e8748ff863449bc37109b85" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46741 } }
I20260620 23:26:32.831033 20729 raft_consensus.cc:374] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:32.831174 20729 raft_consensus.cc:729] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa2be09cc29404baf9be9c207b59d7c, State: Initialized, Role: FOLLOWER
I20260620 23:26:32.831923 20710 heartbeater.cc:503] Master 127.0.97.62:38533 was elected leader, sending a full tablet report...
I20260620 23:26:32.831925 20729 consensus_queue.cc:260] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a34844dfb246caad6b4c66c247f0b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40149 } } peers { permanent_uuid: "cfa2be09cc29404baf9be9c207b59d7c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46157 } } peers { permanent_uuid: "f5a89a1b1e8748ff863449bc37109b85" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46741 } }
I20260620 23:26:32.832647 20728 ts_tablet_manager.cc:1428] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 23:26:32.840294 20729 ts_tablet_manager.cc:1428] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260620 23:26:32.840585 20731 raft_consensus.cc:348] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a34844dfb246caad6b4c66c247f0b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40149 } } peers { permanent_uuid: "cfa2be09cc29404baf9be9c207b59d7c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46157 } } peers { permanent_uuid: "f5a89a1b1e8748ff863449bc37109b85" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46741 } }
I20260620 23:26:32.841006 20731 raft_consensus.cc:374] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:32.841135 20731 raft_consensus.cc:729] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a89a1b1e8748ff863449bc37109b85, State: Initialized, Role: FOLLOWER
I20260620 23:26:32.841784 20731 consensus_queue.cc:260] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a34844dfb246caad6b4c66c247f0b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40149 } } peers { permanent_uuid: "cfa2be09cc29404baf9be9c207b59d7c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46157 } } peers { permanent_uuid: "f5a89a1b1e8748ff863449bc37109b85" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46741 } }
I20260620 23:26:32.843772 20580 heartbeater.cc:503] Master 127.0.97.62:38533 was elected leader, sending a full tablet report...
I20260620 23:26:32.844159 20731 ts_tablet_manager.cc:1428] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
I20260620 23:26:32.913616 20736 raft_consensus.cc:482] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:26:32.914081 20736 raft_consensus.cc:504] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a34844dfb246caad6b4c66c247f0b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40149 } } peers { permanent_uuid: "cfa2be09cc29404baf9be9c207b59d7c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46157 } } peers { permanent_uuid: "f5a89a1b1e8748ff863449bc37109b85" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46741 } }
I20260620 23:26:32.916121 20736 leader_election.cc:288] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40a34844dfb246caad6b4c66c247f0b4 (127.0.97.3:40149), cfa2be09cc29404baf9be9c207b59d7c (127.0.97.1:46157)
I20260620 23:26:32.923514 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.916677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48125 (local address 127.0.97.1:46157)
0620 23:26:32.916972 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:32.916982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:32.917441 (+ 459us) server_negotiation.cc:407] Connection header received
0620 23:26:32.917655 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:32.917661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:32.917747 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:32.917853 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:32.918825 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.919714 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.921721 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.922542 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.922718 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.922833 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:32.923124 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 23:26:32.923264 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 23:26:32.923637 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.916586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59051 (local address 127.0.97.3:40149)
0620 23:26:32.916803 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:32.916810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:32.917445 (+ 635us) server_negotiation.cc:407] Connection header received
0620 23:26:32.917655 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:32.917661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:32.917747 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:32.917853 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:26:32.918664 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.919706 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.921829 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.922627 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.922840 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.922945 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:32.923270 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:26:32.923398 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:26:32.923681 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.916551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46157 (local address 127.0.97.2:48125)
0620 23:26:32.917279 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:32.917304 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:32.917509 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:32.917892 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:32.917897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:32.917935 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:32.918591 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.918604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.919863 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.919869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.921501 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.921516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.922790 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.922796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.922955 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.922980 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 23:26:32.923097 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:26:32.923235 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":144,"spinlock_wait_cycles":50816,"thread_start_us":432,"threads_started":1}
I20260620 23:26:32.923779 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.916574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40149 (local address 127.0.97.2:59051)
0620 23:26:32.917300 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:32.917315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:26:32.917476 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:26:32.917892 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:26:32.917897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:26:32.917925 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:26:32.918477 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.918492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.919846 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.919852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.921624 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:26:32.921638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.922882 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.922888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:26:32.923064 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.923093 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:26:32.923241 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 23:26:32.923371 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":153,"thread_start_us":426,"threads_started":1}
I20260620 23:26:32.924863 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2b3bbb03c8431f8f276130432a1084" candidate_uuid: "f5a89a1b1e8748ff863449bc37109b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a34844dfb246caad6b4c66c247f0b4" is_pre_election: true
I20260620 23:26:32.924863 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2b3bbb03c8431f8f276130432a1084" candidate_uuid: "f5a89a1b1e8748ff863449bc37109b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa2be09cc29404baf9be9c207b59d7c" is_pre_election: true
I20260620 23:26:32.925396 20665 raft_consensus.cc:2454] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5a89a1b1e8748ff863449bc37109b85 in term 0.
I20260620 23:26:32.925395 20404 raft_consensus.cc:2454] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5a89a1b1e8748ff863449bc37109b85 in term 0.
I20260620 23:26:32.926111 20468 leader_election.cc:302] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 40a34844dfb246caad6b4c66c247f0b4, cfa2be09cc29404baf9be9c207b59d7c, f5a89a1b1e8748ff863449bc37109b85; no voters:
I20260620 23:26:32.926518 20736 raft_consensus.cc:2781] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:26:32.926707 20736 raft_consensus.cc:482] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:26:32.926807 20736 raft_consensus.cc:3037] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:32.929109 20736 raft_consensus.cc:504] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a34844dfb246caad6b4c66c247f0b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40149 } } peers { permanent_uuid: "cfa2be09cc29404baf9be9c207b59d7c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46157 } } peers { permanent_uuid: "f5a89a1b1e8748ff863449bc37109b85" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46741 } }
I20260620 23:26:32.929764 20736 leader_election.cc:288] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [CANDIDATE]: Term 1 election: Requested vote from peers 40a34844dfb246caad6b4c66c247f0b4 (127.0.97.3:40149), cfa2be09cc29404baf9be9c207b59d7c (127.0.97.1:46157)
I20260620 23:26:32.930245 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2b3bbb03c8431f8f276130432a1084" candidate_uuid: "f5a89a1b1e8748ff863449bc37109b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a34844dfb246caad6b4c66c247f0b4"
I20260620 23:26:32.930299 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2b3bbb03c8431f8f276130432a1084" candidate_uuid: "f5a89a1b1e8748ff863449bc37109b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa2be09cc29404baf9be9c207b59d7c"
I20260620 23:26:32.930473 20665 raft_consensus.cc:3037] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:32.930511 20404 raft_consensus.cc:3037] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:32.932806 20665 raft_consensus.cc:2454] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5a89a1b1e8748ff863449bc37109b85 in term 1.
I20260620 23:26:32.932812 20404 raft_consensus.cc:2454] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5a89a1b1e8748ff863449bc37109b85 in term 1.
I20260620 23:26:32.933290 20468 leader_election.cc:302] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40a34844dfb246caad6b4c66c247f0b4, f5a89a1b1e8748ff863449bc37109b85; no voters:
I20260620 23:26:32.933624 20736 raft_consensus.cc:2781] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:32.934127 20736 raft_consensus.cc:686] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 1 LEADER]: Becoming Leader. State: Replica: f5a89a1b1e8748ff863449bc37109b85, State: Running, Role: LEADER
I20260620 23:26:32.934715 20736 consensus_queue.cc:237] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a34844dfb246caad6b4c66c247f0b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40149 } } peers { permanent_uuid: "cfa2be09cc29404baf9be9c207b59d7c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46157 } } peers { permanent_uuid: "f5a89a1b1e8748ff863449bc37109b85" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46741 } }
I20260620 23:26:32.939327 20263 catalog_manager.cc:5495] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5a89a1b1e8748ff863449bc37109b85 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f5a89a1b1e8748ff863449bc37109b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a34844dfb246caad6b4c66c247f0b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfa2be09cc29404baf9be9c207b59d7c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5a89a1b1e8748ff863449bc37109b85" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46741 } health_report { overall_health: HEALTHY } } }
W20260620 23:26:32.952946 20711 tablet.cc:2369] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:26:32.970335 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:32.964859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.0.97.2:46741)
0620 23:26:32.965076 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:32.965082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:32.965102 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:26:32.965353 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:32.965357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:32.965448 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:32.965620 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:26:32.965973 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.966755 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.968155 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:32.968579 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:32.968728 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:32.969227 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:26:32.969750 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:26:32.969924 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:32.969976 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 23:26:32.970059 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 23:26:32.970929 20450 tablet.cc:2369] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:26:32.977406 20581 tablet.cc:2369] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:26:32.979010 20665 raft_consensus.cc:1261] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4 [term 1 FOLLOWER]: Refusing update from remote peer f5a89a1b1e8748ff863449bc37109b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:26:32.979060 20404 raft_consensus.cc:1261] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c [term 1 FOLLOWER]: Refusing update from remote peer f5a89a1b1e8748ff863449bc37109b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:26:32.980050 20739 consensus_queue.cc:1035] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40a34844dfb246caad6b4c66c247f0b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:26:32.980446 20736 consensus_queue.cc:1035] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfa2be09cc29404baf9be9c207b59d7c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:26:32.991972 20742 mvcc.cc:205] Tried to move back new op lower bound from 7299063779231322112 to 7299063779065061376. Current Snapshot: MvccSnapshot[applied={T|T < 7299063779231322112}]
I20260620 23:26:33.005354 20262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:26:33.005656 20262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:26:33.009548 20262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e657b257e40d4a8bb4f0c620bf9b35d0: Sending DeleteTablet for 3 replicas of tablet fe2b3bbb03c8431f8f276130432a1084
I20260620 23:26:33.010965 20515 tablet_service.cc:1514] Processing DeleteTablet for tablet fe2b3bbb03c8431f8f276130432a1084 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:33 UTC) from {username='slave'} at 127.0.0.1:41636
I20260620 23:26:33.011281 20384 tablet_service.cc:1514] Processing DeleteTablet for tablet fe2b3bbb03c8431f8f276130432a1084 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:33 UTC) from {username='slave'} at 127.0.0.1:39492
I20260620 23:26:33.011317 20645 tablet_service.cc:1514] Processing DeleteTablet for tablet fe2b3bbb03c8431f8f276130432a1084 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:26:33 UTC) from {username='slave'} at 127.0.0.1:47554
I20260620 23:26:33.013288 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 20322
I20260620 23:26:33.026722 20754 tablet_replica.cc:330] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4: stopping tablet replica
I20260620 23:26:33.027438 20754 raft_consensus.cc:2229] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:26:33.028196 20754 raft_consensus.cc:2258] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:33.031780 20754 ts_tablet_manager.cc:1905] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:33.040447 20754 ts_tablet_manager.cc:1918] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:33.040755 20754 log.cc:1196] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-2/wal/wals/fe2b3bbb03c8431f8f276130432a1084
I20260620 23:26:33.041401 20754 ts_tablet_manager.cc:1939] T fe2b3bbb03c8431f8f276130432a1084 P 40a34844dfb246caad6b4c66c247f0b4: Deleting consensus metadata
I20260620 23:26:33.043609 20247 catalog_manager.cc:4841] TS 40a34844dfb246caad6b4c66c247f0b4 (127.0.97.3:40149): tablet fe2b3bbb03c8431f8f276130432a1084 (table test-table [id=56313ea655cb4b14a389e578575e3ca3]) successfully deleted
I20260620 23:26:33.044260 20753 tablet_replica.cc:330] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c: stopping tablet replica
I20260620 23:26:33.044749 20752 tablet_replica.cc:330] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85: stopping tablet replica
I20260620 23:26:33.044912 20753 raft_consensus.cc:2229] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:26:33.045233 20752 raft_consensus.cc:2229] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:26:33.045588 20753 raft_consensus.cc:2258] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:33.045934 20752 raft_consensus.cc:2258] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:26:33.048074 20753 ts_tablet_manager.cc:1905] T fe2b3bbb03c8431f8f276130432a1084 P cfa2be09cc29404baf9be9c207b59d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:33.048084 20752 ts_tablet_manager.cc:1905] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:26:33.050634 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:33.014906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.0.97.1:46157)
0620 23:26:33.015132 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:33.015141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:33.039185 (+ 24044us) server_negotiation.cc:407] Connection header received
0620 23:26:33.039692 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:33.039699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:33.039826 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:33.039899 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:33.040666 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:33.041866 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:33.044624 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:33.045353 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:33.045466 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:33.048756 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:33.048770 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:33.048788 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:33.049599 (+ 811us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:33.049665 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:33.049701 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:33.049810 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:33.050088 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 23:26:33.050210 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:26:33.052068 20444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:33.058120 20752 ts_tablet_manager.cc:1918] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:26:33.058317 20752 log.cc:1196] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85: Deleting WAL directory at /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997680534866-388-0/minicluster-data/ts-1/wal/wals/fe2b3bbb03c8431f8f276130432a1084
I20260620 23:26:33.058934 20752 ts_tablet_manager.cc:1939] T fe2b3bbb03c8431f8f276130432a1084 P f5a89a1b1e8748ff863449bc37109b85: Deleting consensus metadata
I20260620 23:26:33.060978 20250 catalog_manager.cc:4841] TS f5a89a1b1e8748ff863449bc37109b85 (127.0.97.2:46741): tablet fe2b3bbb03c8431f8f276130432a1084 (table test-table [id=56313ea655cb4b14a389e578575e3ca3]) successfully deleted
I20260620 23:26:33.144307 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 20322
W20260620 23:26:33.168558 20249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46157 (error 108)
I20260620 23:26:33.168790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 20453
I20260620 23:26:33.171248 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:33.170757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46157 (local address 127.0.0.1:39498)
0620 23:26:33.170932 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:33.171062 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 23:26:33.171543 20249 catalog_manager.cc:4584] TS cfa2be09cc29404baf9be9c207b59d7c (127.0.97.1:46157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe2b3bbb03c8431f8f276130432a1084: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46157: connect: Connection refused (error 111)
I20260620 23:26:33.177618 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:33.169976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.0.97.2:46741)
0620 23:26:33.170251 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:33.170260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:33.170284 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:26:33.170355 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:33.170360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:33.170473 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:33.170606 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:33.171136 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:33.172172 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:33.174097 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:33.174598 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:33.174785 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:33.176037 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:33.176048 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:33.176064 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:33.176725 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:33.176776 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:33.176798 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:33.176914 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:33.177112 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 23:26:33.177204 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260620 23:26:33.178346 20575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:33.225302 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:33.224744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46157 (local address 127.0.0.1:39514)
0620 23:26:33.224985 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:33.225098 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:26:33.226804 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:33.226319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46157 (local address 127.0.0.1:39516)
0620 23:26:33.226500 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:33.226605 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 23:26:33.287969 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:33.287336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46157 (local address 127.0.0.1:39520)
0620 23:26:33.287630 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:33.287747 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 23:26:33.289109 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:33.288717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46157 (local address 127.0.0.1:39524)
0620 23:26:33.288880 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:33.288961 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:26:33.293009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 20453
I20260620 23:26:33.312062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 20583
I20260620 23:26:33.319617 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:33.312899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.0.97.3:40149)
0620 23:26:33.313120 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:33.313126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:33.313147 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:26:33.313353 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:33.313358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:33.313440 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:33.313563 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:33.313949 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:33.314762 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:33.316071 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:33.316479 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:33.316618 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:33.318091 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:33.318103 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:33.318118 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:33.318810 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:33.318864 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:33.318885 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:33.319015 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:33.319161 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 23:26:33.319277 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:26:33.320243 20705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:33.370090 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:33.369612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46157 (local address 127.0.0.1:39528)
0620 23:26:33.369833 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:33.369929 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 23:26:33.371183 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:33.370783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46157 (local address 127.0.0.1:39540)
0620 23:26:33.370966 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:26:33.371074 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:26:33.409415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 20583
I20260620 23:26:33.427306 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 20232
I20260620 23:26:33.428412 20293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:33.539222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 20232
2026-06-20T23:26:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2963 ms)
[----------] 6 tests from EncryptionPolicyTest (22527 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:33 dist-test-slave-5kzz krb5kdc[20763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:33 dist-test-slave-5kzz krb5kdc[20763](info): set up 2 sockets
Jun 20 23:26:33 dist-test-slave-5kzz krb5kdc[20763](info): commencing operation
krb5kdc: starting...
W20260620 23:26:34.683202 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.006s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:34 dist-test-slave-5kzz krb5kdc[20763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:26:34.947927 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:35.302719 20776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:35.303041 20776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:35.303138 20776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:35.310746 20776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:35.310879 20776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:35.310935 20776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:35.310989 20776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:35.311036 20776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:35.321087 20776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:35.322387 20776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:35.323721 20776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:35.336202 20781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:35.336117 20782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:35.336179 20784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:35.340248 20776 server_base.cc:1034] running on GCE node
I20260620 23:26:35.341116 20776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:35.342730 20776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:35.343971 20776 hybrid_clock.cc:648] HybridClock initialized: now 1781997995343930 us; error 39 us; skew 500 ppm
I20260620 23:26:35.344332 20776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:35 dist-test-slave-5kzz krb5kdc[20763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:35.347512 20776 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:26:35.349043 20776 webserver.cc:460] Webserver started at http://127.0.0.1:42863/ using document root <none> and password file <none>
I20260620 23:26:35.349704 20776 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:35.349800 20776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:35.350061 20776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:35.353008 20776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "5c4e53ba04d449a98aee7e393ca4d9a1"
format_stamp: "Formatted at 2026-06-20 23:26:35 on dist-test-slave-5kzz"
server_key: "08cfd8489c01c6d18b02206fb8eea2cd"
server_key_iv: "bb4c580838052b4847b5a1394d684e51"
server_key_version: "encryptionkey@0"
I20260620 23:26:35.353744 20776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "5c4e53ba04d449a98aee7e393ca4d9a1"
format_stamp: "Formatted at 2026-06-20 23:26:35 on dist-test-slave-5kzz"
server_key: "08cfd8489c01c6d18b02206fb8eea2cd"
server_key_iv: "bb4c580838052b4847b5a1394d684e51"
server_key_version: "encryptionkey@0"
I20260620 23:26:35.359027 20776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 23:26:35.362536 20791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:35.364135 20776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:26:35.364336 20776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "5c4e53ba04d449a98aee7e393ca4d9a1"
format_stamp: "Formatted at 2026-06-20 23:26:35 on dist-test-slave-5kzz"
server_key: "08cfd8489c01c6d18b02206fb8eea2cd"
server_key_iv: "bb4c580838052b4847b5a1394d684e51"
server_key_version: "encryptionkey@0"
I20260620 23:26:35.364550 20776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:35.399425 20776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:35.404016 20776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:35.404331 20776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:35.423931 20776 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40527
I20260620 23:26:35.424045 20842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40527 every 8 connection(s)
I20260620 23:26:35.425449 20776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:35.429649 20843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:35.435642 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 20776
I20260620 23:26:35.435885 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:35.436533 388 external_mini_cluster.cc:1428] Setting key 22e5f262b62becfba1280a4592c488e7
I20260620 23:26:35.442013 20843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1: Bootstrap starting.
I20260620 23:26:35.445272 20843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:35.446327 20843 log.cc:824] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:35.449427 20843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1: No bootstrap required, opened a new log
Jun 20 23:26:35 dist-test-slave-5kzz krb5kdc[20763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:26:35.457944 20843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e53ba04d449a98aee7e393ca4d9a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } }
I20260620 23:26:35.458359 20843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:35.458494 20843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4e53ba04d449a98aee7e393ca4d9a1, State: Initialized, Role: FOLLOWER
I20260620 23:26:35.459313 20843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e53ba04d449a98aee7e393ca4d9a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } }
I20260620 23:26:35.459592 20843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:35.459758 20843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:35.459973 20843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:35.460639 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:35.438980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59107 (local address 127.0.0.1:40527)
0620 23:26:35.439887 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:35.439900 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:35.439944 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:26:35.441697 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:35.441720 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:35.442223 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:35.442640 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:35.444192 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:35.445541 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:35.446987 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:35.447741 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:35.447945 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:35.452438 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:35.452446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:35.452464 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:35.452507 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:35.455955 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:35.456866 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:35.456876 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:35.456886 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:35.457059 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:35.457550 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:35.457554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:35.457556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:35.458099 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:35.458460 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:35.458982 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 23:26:35.459301 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":467,"threads_started":1}
I20260620 23:26:35.462668 20843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e53ba04d449a98aee7e393ca4d9a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } }
I20260620 23:26:35.463387 20843 leader_election.cc:302] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c4e53ba04d449a98aee7e393ca4d9a1; no voters:
I20260620 23:26:35.464144 20843 leader_election.cc:288] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:35.464293 20848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:35.465101 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:35.466188 20843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:35.467097 20848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [term 1 LEADER]: Becoming Leader. State: Replica: 5c4e53ba04d449a98aee7e393ca4d9a1, State: Running, Role: LEADER
I20260620 23:26:35.467780 20848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e53ba04d449a98aee7e393ca4d9a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } }
I20260620 23:26:35.471153 20850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c4e53ba04d449a98aee7e393ca4d9a1. Latest consensus state: current_term: 1 leader_uuid: "5c4e53ba04d449a98aee7e393ca4d9a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e53ba04d449a98aee7e393ca4d9a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } } }
I20260620 23:26:35.471655 20850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:35.472363 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c4e53ba04d449a98aee7e393ca4d9a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e53ba04d449a98aee7e393ca4d9a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } } }
I20260620 23:26:35.472747 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:35.477598 20855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:35.484297 20855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:35.506129 20855 catalog_manager.cc:1347] Generated new cluster ID: d958738463b04cebb921e9c0ad3c1544
I20260620 23:26:35.506265 20855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:35.508879 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:35.481208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41031 (local address 127.0.0.1:40527)
0620 23:26:35.481389 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:35.481395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:35.486039 (+ 4644us) server_negotiation.cc:407] Connection header received
0620 23:26:35.486259 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:35.486266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:35.486357 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:35.486473 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:35.489162 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:35.490855 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:35.493177 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:35.494151 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:35.494314 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:35.498785 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:35.498796 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:35.498805 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:35.498864 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:35.503681 (+ 4817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:35.505123 (+ 1442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:35.505131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:35.505138 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:35.505370 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:35.506134 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:35.506145 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:35.506151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:35.506770 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:35.507155 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:35.508040 (+ 885us) server_negotiation.cc:299] Negotiation successful
0620 23:26:35.508322 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:26:35.536934 20855 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:35.538496 20855 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:35.547852 20855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c4e53ba04d449a98aee7e393ca4d9a1: Generated new TSK 0
I20260620 23:26:35.549072 20855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:35.560268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 20776
I20260620 23:26:35.561692 20838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:35.658084 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 20776
W20260620 23:26:35.675894 20761 connection.cc:535] client connection to 127.0.0.1:40527 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40527): Connection reset by peer (error 104)
2026-06-20T23:26:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2111 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:35 dist-test-slave-5kzz krb5kdc[20875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:35 dist-test-slave-5kzz krb5kdc[20875](info): set up 2 sockets
Jun 20 23:26:35 dist-test-slave-5kzz krb5kdc[20875](info): commencing operation
krb5kdc: starting...
W20260620 23:26:36.783102 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:36 dist-test-slave-5kzz krb5kdc[20875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:26:37.053647 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:37.398198 20888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:37.398535 20888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:37.398746 20888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:37.406098 20888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:37.406222 20888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:37.406311 20888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:37.406433 20888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:37.406514 20888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:37.416548 20888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40355
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:37.417912 20888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:37.419240 20888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:37.431460 20894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:37.431499 20896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:37.431531 20893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:37.432967 20888 server_base.cc:1034] running on GCE node
I20260620 23:26:37.433818 20888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:37.435380 20888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:37.436600 20888 hybrid_clock.cc:648] HybridClock initialized: now 1781997997436546 us; error 41 us; skew 500 ppm
I20260620 23:26:37.436972 20888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:37 dist-test-slave-5kzz krb5kdc[20875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:37.444399 20888 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:26:37.446029 20888 webserver.cc:460] Webserver started at http://127.0.0.1:46047/ using document root <none> and password file <none>
I20260620 23:26:37.446789 20888 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:37.446926 20888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:37.447255 20888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:37.450275 20888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "8f1721edf5dd4f7f8641f2586b25d0c3"
format_stamp: "Formatted at 2026-06-20 23:26:37 on dist-test-slave-5kzz"
server_key: "7c06989aef3c63c378aeb017683bd31c"
server_key_iv: "ce34a878001a3816ded629cece48e33e"
server_key_version: "encryptionkey@0"
I20260620 23:26:37.451071 20888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "8f1721edf5dd4f7f8641f2586b25d0c3"
format_stamp: "Formatted at 2026-06-20 23:26:37 on dist-test-slave-5kzz"
server_key: "7c06989aef3c63c378aeb017683bd31c"
server_key_iv: "ce34a878001a3816ded629cece48e33e"
server_key_version: "encryptionkey@0"
I20260620 23:26:37.456197 20888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 23:26:37.459817 20903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:37.461565 20888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:26:37.461781 20888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "8f1721edf5dd4f7f8641f2586b25d0c3"
format_stamp: "Formatted at 2026-06-20 23:26:37 on dist-test-slave-5kzz"
server_key: "7c06989aef3c63c378aeb017683bd31c"
server_key_iv: "ce34a878001a3816ded629cece48e33e"
server_key_version: "encryptionkey@0"
I20260620 23:26:37.462010 20888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:37.481375 20888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:37.485391 20888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:37.485688 20888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:37.504604 20888 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40355
I20260620 23:26:37.504649 20954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40355 every 8 connection(s)
I20260620 23:26:37.506016 20888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:37.509800 20955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:37.510667 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 20888
I20260620 23:26:37.510885 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:37.511298 388 external_mini_cluster.cc:1428] Setting key 562cb2b0c51649e952849a3d4211f936
I20260620 23:26:37.522570 20955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3: Bootstrap starting.
I20260620 23:26:37.525018 20958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:37.513230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59217 (local address 127.0.0.1:40355)
0620 23:26:37.514000 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:37.514012 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:37.514053 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:26:37.516252 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:37.516276 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:37.516704 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:37.517138 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:37.517669 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:37.518989 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:37.520348 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:37.521038 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:37.521239 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:37.522621 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:37.522628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:37.522643 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:37.523313 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:37.523372 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:37.523404 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:37.523574 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:37.523881 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 23:26:37.524072 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":343,"threads_started":1}
I20260620 23:26:37.525930 20955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:37.527067 20955 log.cc:824] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:37.528950 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:37.530577 20955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3: No bootstrap required, opened a new log
I20260620 23:26:37.541118 20958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:37.534398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59133 (local address 127.0.0.1:40355)
0620 23:26:37.534588 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:37.534595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:37.534929 (+ 334us) server_negotiation.cc:407] Connection header received
0620 23:26:37.535093 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:37.535099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:37.535188 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:37.535311 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:37.535784 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:37.536998 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:37.538342 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:37.539104 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:37.539241 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:37.540367 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:37.540373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:37.540377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:37.540401 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:37.540443 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:37.540462 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:37.540578 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:37.540715 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:26:37.540773 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:26:37.541236 20955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1721edf5dd4f7f8641f2586b25d0c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40355 } }
I20260620 23:26:37.541689 20955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:37.541839 20955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1721edf5dd4f7f8641f2586b25d0c3, State: Initialized, Role: FOLLOWER
I20260620 23:26:37.542450 20955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1721edf5dd4f7f8641f2586b25d0c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40355 } }
I20260620 23:26:37.542740 20955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:37.542881 20955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:37.543067 20955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:37.545292 20955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1721edf5dd4f7f8641f2586b25d0c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40355 } }
I20260620 23:26:37.545822 20955 leader_election.cc:302] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f1721edf5dd4f7f8641f2586b25d0c3; no voters:
I20260620 23:26:37.546399 20955 leader_election.cc:288] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:37.546546 20965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:37.547952 20965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [term 1 LEADER]: Becoming Leader. State: Replica: 8f1721edf5dd4f7f8641f2586b25d0c3, State: Running, Role: LEADER
I20260620 23:26:37.548767 20955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:37.548709 20965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1721edf5dd4f7f8641f2586b25d0c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40355 } }
I20260620 23:26:37.551700 20966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f1721edf5dd4f7f8641f2586b25d0c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1721edf5dd4f7f8641f2586b25d0c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40355 } } }
I20260620 23:26:37.551726 20967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f1721edf5dd4f7f8641f2586b25d0c3. Latest consensus state: current_term: 1 leader_uuid: "8f1721edf5dd4f7f8641f2586b25d0c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1721edf5dd4f7f8641f2586b25d0c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40355 } } }
I20260620 23:26:37.552155 20967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:37.552151 20966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:37.553094 20975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:37.560997 20975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:37.572840 20975 catalog_manager.cc:1347] Generated new cluster ID: 08fda5fe32ee4eb49a9de609c4c87d8c
I20260620 23:26:37.572979 20975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:37.605645 20975 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:37.607123 20975 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:37.617149 20975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f1721edf5dd4f7f8641f2586b25d0c3: Generated new TSK 0
I20260620 23:26:37.618096 20975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:37.632823 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 20888
W20260620 23:26:37.634048 20950 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59217
W20260620 23:26:37.634981 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 20888: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:26:37.635186 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 20888
W20260620 23:26:37.652341 20873 connection.cc:535] client connection to 127.0.0.1:40355 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40355): Connection reset by peer (error 104)
2026-06-20T23:26:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:37 dist-test-slave-5kzz krb5kdc[20986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:37 dist-test-slave-5kzz krb5kdc[20986](info): set up 2 sockets
Jun 20 23:26:37 dist-test-slave-5kzz krb5kdc[20986](info): commencing operation
krb5kdc: starting...
W20260620 23:26:38.755059 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:38 dist-test-slave-5kzz krb5kdc[20986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:26:39.029497 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:39.390492 20999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:39.390938 20999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:39.391106 20999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:39.399008 20999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:39.399176 20999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:39.399295 20999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:39.399401 20999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:39.399480 20999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:39.410082 20999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:39.411576 20999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:39.412957 20999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:39.425325 21005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:39.425309 21007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:39.425398 21004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:39.426139 20999 server_base.cc:1034] running on GCE node
I20260620 23:26:39.426992 20999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:39.428474 20999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:39.429706 20999 hybrid_clock.cc:648] HybridClock initialized: now 1781997999429654 us; error 46 us; skew 500 ppm
I20260620 23:26:39.430065 20999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:39 dist-test-slave-5kzz krb5kdc[20986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:39.433622 20999 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:26:39.435526 20999 webserver.cc:460] Webserver started at http://127.0.0.1:45481/ using document root <none> and password file <none>
I20260620 23:26:39.436239 20999 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:39.436347 20999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:39.436645 20999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:39.439958 20999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "7e46d094c6124880b109b485fda27ee6"
format_stamp: "Formatted at 2026-06-20 23:26:39 on dist-test-slave-5kzz"
server_key: "901ef7583d1875c4816dce3461dbc3b6"
server_key_iv: "70bcaa4c0b0d0baf015c803bb829f1c8"
server_key_version: "encryptionkey@0"
I20260620 23:26:39.440810 20999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "7e46d094c6124880b109b485fda27ee6"
format_stamp: "Formatted at 2026-06-20 23:26:39 on dist-test-slave-5kzz"
server_key: "901ef7583d1875c4816dce3461dbc3b6"
server_key_iv: "70bcaa4c0b0d0baf015c803bb829f1c8"
server_key_version: "encryptionkey@0"
I20260620 23:26:39.446650 20999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:26:39.450631 21014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:39.452315 20999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:26:39.452560 20999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "7e46d094c6124880b109b485fda27ee6"
format_stamp: "Formatted at 2026-06-20 23:26:39 on dist-test-slave-5kzz"
server_key: "901ef7583d1875c4816dce3461dbc3b6"
server_key_iv: "70bcaa4c0b0d0baf015c803bb829f1c8"
server_key_version: "encryptionkey@0"
I20260620 23:26:39.452769 20999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:39.459086 20999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:39.463892 20999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:39.464233 20999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:39.484102 20999 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36281
I20260620 23:26:39.484150 21065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36281 every 8 connection(s)
I20260620 23:26:39.485591 20999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:39.486719 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 20999
I20260620 23:26:39.486900 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:39.487272 388 external_mini_cluster.cc:1428] Setting key ba34dd7217325feeab47e41e4bf1e99c
I20260620 23:26:39.490381 21066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:39.496251 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:39.489162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46535 (local address 127.0.0.1:36281)
0620 23:26:39.489983 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:39.489995 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:39.490037 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:26:39.492349 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:39.492374 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:39.492764 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:39.493125 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:39.494229 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:39.494238 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:39.494259 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:39.494280 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:39.494351 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:39.494374 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:39.494639 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:39.494990 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 23:26:39.495227 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":352,"threads_started":1}
I20260620 23:26:39.500356 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:39.505108 21066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6: Bootstrap starting.
I20260620 23:26:39.508515 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:39.505795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42377 (local address 127.0.0.1:36281)
0620 23:26:39.505974 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:39.505980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:39.506380 (+ 400us) server_negotiation.cc:407] Connection header received
0620 23:26:39.506508 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:39.506512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:39.506581 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:39.506725 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:39.507821 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:39.507827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:39.507831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:39.507849 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:39.507900 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:39.507921 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:39.508044 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:39.508157 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 23:26:39.508220 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:26:39.509130 21066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:39.510552 21066 log.cc:824] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:39.514348 21066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6: No bootstrap required, opened a new log
I20260620 23:26:39.523018 21066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e46d094c6124880b109b485fda27ee6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260620 23:26:39.523509 21066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:39.523622 21066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e46d094c6124880b109b485fda27ee6, State: Initialized, Role: FOLLOWER
I20260620 23:26:39.524323 21066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e46d094c6124880b109b485fda27ee6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260620 23:26:39.524636 21066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:39.524870 21066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:39.525069 21066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:39.527423 21066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e46d094c6124880b109b485fda27ee6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260620 23:26:39.527933 21066 leader_election.cc:302] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e46d094c6124880b109b485fda27ee6; no voters:
I20260620 23:26:39.528606 21066 leader_election.cc:288] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:39.528950 21076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:39.529834 21076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [term 1 LEADER]: Becoming Leader. State: Replica: 7e46d094c6124880b109b485fda27ee6, State: Running, Role: LEADER
I20260620 23:26:39.530550 21076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e46d094c6124880b109b485fda27ee6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260620 23:26:39.531009 21066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:39.533846 21077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e46d094c6124880b109b485fda27ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e46d094c6124880b109b485fda27ee6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } } }
I20260620 23:26:39.533648 21078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e46d094c6124880b109b485fda27ee6. Latest consensus state: current_term: 1 leader_uuid: "7e46d094c6124880b109b485fda27ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e46d094c6124880b109b485fda27ee6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } } }
I20260620 23:26:39.534192 21077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:39.534240 21078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:39.537009 21084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:39.542662 21084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:39.552910 21084 catalog_manager.cc:1347] Generated new cluster ID: 944b74a4330a4cbc9ebde6195bfcb636
I20260620 23:26:39.553071 21084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:39.570487 21084 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:39.571120 21084 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:39.580967 21084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e46d094c6124880b109b485fda27ee6: Generated new TSK 0
I20260620 23:26:39.582122 21084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:39.588495 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 20999
W20260620 23:26:39.589808 21061 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46535
W20260620 23:26:39.590725 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 20999: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:26:39.590891 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 20999
2026-06-20T23:26:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:39 dist-test-slave-5kzz krb5kdc[21097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:39 dist-test-slave-5kzz krb5kdc[21097](info): set up 2 sockets
Jun 20 23:26:39 dist-test-slave-5kzz krb5kdc[21097](info): commencing operation
krb5kdc: starting...
W20260620 23:26:40.707120 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:40 dist-test-slave-5kzz krb5kdc[21097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:26:40.975479 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:41.316399 21110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:41.316704 21110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:41.316802 21110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:41.324139 21110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:41.324255 21110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:41.324302 21110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:41.324365 21110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:41.324409 21110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:41.334285 21110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:41.335625 21110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:41.336850 21110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:41.348843 21115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:41.348871 21118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:41.348937 21116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:41.352350 21110 server_base.cc:1034] running on GCE node
I20260620 23:26:41.353157 21110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:41.354656 21110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:41.355981 21110 hybrid_clock.cc:648] HybridClock initialized: now 1781998001355935 us; error 39 us; skew 500 ppm
I20260620 23:26:41.356395 21110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:41 dist-test-slave-5kzz krb5kdc[21097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:41.359618 21110 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:26:41.361251 21110 webserver.cc:460] Webserver started at http://127.0.0.1:42607/ using document root <none> and password file <none>
I20260620 23:26:41.361980 21110 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:41.362120 21110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:41.362429 21110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:41.365459 21110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "f69f39b2990242a18ad8c0b9f47c3f9a"
format_stamp: "Formatted at 2026-06-20 23:26:41 on dist-test-slave-5kzz"
server_key: "f5a5b0e4c2d2cbcc2938f5e44d064bc1"
server_key_iv: "a83fe3b2564b374c3d1ac7f9228aba9a"
server_key_version: "encryptionkey@0"
I20260620 23:26:41.366231 21110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "f69f39b2990242a18ad8c0b9f47c3f9a"
format_stamp: "Formatted at 2026-06-20 23:26:41 on dist-test-slave-5kzz"
server_key: "f5a5b0e4c2d2cbcc2938f5e44d064bc1"
server_key_iv: "a83fe3b2564b374c3d1ac7f9228aba9a"
server_key_version: "encryptionkey@0"
I20260620 23:26:41.371582 21110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:26:41.375175 21125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:41.376549 21110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:26:41.376770 21110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "f69f39b2990242a18ad8c0b9f47c3f9a"
format_stamp: "Formatted at 2026-06-20 23:26:41 on dist-test-slave-5kzz"
server_key: "f5a5b0e4c2d2cbcc2938f5e44d064bc1"
server_key_iv: "a83fe3b2564b374c3d1ac7f9228aba9a"
server_key_version: "encryptionkey@0"
I20260620 23:26:41.376981 21110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:41.394605 21110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:41.399286 21110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:41.399592 21110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:41.419832 21110 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42051
I20260620 23:26:41.419858 21176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42051 every 8 connection(s)
I20260620 23:26:41.421360 21110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:41.421787 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 21110
I20260620 23:26:41.421975 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:41.422811 388 external_mini_cluster.cc:1428] Setting key df8f9acee8f8e1e60312dfce672c61eb
I20260620 23:26:41.426460 21177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:26:41 dist-test-slave-5kzz krb5kdc[21097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:26:41.439545 21177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a: Bootstrap starting.
I20260620 23:26:41.443958 21177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:41.445258 21177 log.cc:824] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:41.448333 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:41.424744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43473 (local address 127.0.0.1:42051)
0620 23:26:41.425556 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:41.425572 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:41.425621 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:26:41.428012 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:41.428039 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:41.428567 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:41.429032 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:41.430401 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:41.432514 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:41.433911 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:41.435012 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:41.435205 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:41.439429 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:41.439440 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:41.439461 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:41.439510 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:41.443484 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:41.444392 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:41.444408 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:41.444419 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:41.444625 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:41.445019 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:41.445027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:41.445031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:41.445780 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:41.446082 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:41.446549 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 23:26:41.447023 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":328,"threads_started":1}
I20260620 23:26:41.448846 21177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a: No bootstrap required, opened a new log
I20260620 23:26:41.452102 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:41.457304 21177 raft_consensus.cc:348] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69f39b2990242a18ad8c0b9f47c3f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260620 23:26:41.457815 21177 raft_consensus.cc:374] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:41.457928 21177 raft_consensus.cc:729] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69f39b2990242a18ad8c0b9f47c3f9a, State: Initialized, Role: FOLLOWER
I20260620 23:26:41.458648 21177 consensus_queue.cc:260] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69f39b2990242a18ad8c0b9f47c3f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260620 23:26:41.459000 21177 raft_consensus.cc:388] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:41.459149 21177 raft_consensus.cc:482] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:41.459363 21177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:41.462440 21177 raft_consensus.cc:504] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69f39b2990242a18ad8c0b9f47c3f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260620 23:26:41.463120 21177 leader_election.cc:302] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f69f39b2990242a18ad8c0b9f47c3f9a; no voters:
I20260620 23:26:41.463851 21177 leader_election.cc:288] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:41.463909 21187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:41.465303 21187 raft_consensus.cc:686] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [term 1 LEADER]: Becoming Leader. State: Replica: f69f39b2990242a18ad8c0b9f47c3f9a, State: Running, Role: LEADER
I20260620 23:26:41.465932 21187 consensus_queue.cc:237] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69f39b2990242a18ad8c0b9f47c3f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260620 23:26:41.466344 21177 sys_catalog.cc:564] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:41.469352 21189 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f69f39b2990242a18ad8c0b9f47c3f9a. Latest consensus state: current_term: 1 leader_uuid: "f69f39b2990242a18ad8c0b9f47c3f9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69f39b2990242a18ad8c0b9f47c3f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } } }
I20260620 23:26:41.469856 21189 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:41.469453 21188 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f69f39b2990242a18ad8c0b9f47c3f9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69f39b2990242a18ad8c0b9f47c3f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } } }
I20260620 23:26:41.470047 21188 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:41.474877 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:41.456316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60343 (local address 127.0.0.1:42051)
0620 23:26:41.456534 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:41.456543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:41.456985 (+ 442us) server_negotiation.cc:407] Connection header received
0620 23:26:41.457153 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:41.457156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:41.457217 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:41.457299 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:41.459258 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:41.460070 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:41.461836 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:41.462265 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:41.462360 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:41.465510 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:41.465516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:41.465520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:41.465568 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:41.470107 (+ 4539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:41.471383 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:41.471392 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:41.471398 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:41.471601 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:41.472136 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:41.472144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:41.472150 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:41.472675 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:41.473257 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:41.474261 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0620 23:26:41.474506 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:26:41.476612 21194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:41.482175 21194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:41.492151 21194 catalog_manager.cc:1347] Generated new cluster ID: c861a424fcb64966bc92995753b93e56
I20260620 23:26:41.492298 21194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:41.514053 21194 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:41.516052 21194 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:41.524709 21194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f69f39b2990242a18ad8c0b9f47c3f9a: Generated new TSK 0
I20260620 23:26:41.525833 21194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:41.531440 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21110
I20260620 23:26:41.532653 21172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:41.627734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 21110
2026-06-20T23:26:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2036 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:41 dist-test-slave-5kzz krb5kdc[21209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:41 dist-test-slave-5kzz krb5kdc[21209](info): set up 2 sockets
Jun 20 23:26:41 dist-test-slave-5kzz krb5kdc[21209](info): commencing operation
krb5kdc: starting...
W20260620 23:26:42.743008 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:42 dist-test-slave-5kzz krb5kdc[21209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:26:43.005921 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:43.343783 21222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:43.344099 21222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:43.344280 21222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:43.352365 21222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:43.352558 21222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:43.352669 21222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:43.352751 21222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:43.352828 21222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:43.363240 21222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:43.364603 21222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:43.365989 21222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:43.378777 21227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:43.378722 21230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:43.378669 21228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:43.379752 21222 server_base.cc:1034] running on GCE node
I20260620 23:26:43.380656 21222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:43.382270 21222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:43.383525 21222 hybrid_clock.cc:648] HybridClock initialized: now 1781998003383487 us; error 39 us; skew 500 ppm
I20260620 23:26:43.383913 21222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:43 dist-test-slave-5kzz krb5kdc[21209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:43.390496 21222 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:26:43.392277 21222 webserver.cc:460] Webserver started at http://127.0.0.1:42417/ using document root <none> and password file <none>
I20260620 23:26:43.393118 21222 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:43.393240 21222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:43.393518 21222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:43.396780 21222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "6b33d862e0f54733822d0ce15252fa62"
format_stamp: "Formatted at 2026-06-20 23:26:43 on dist-test-slave-5kzz"
server_key: "25842f04ab17c91d87487d420c55b462"
server_key_iv: "3cdb0707a28ddb5a082865cf0360962b"
server_key_version: "encryptionkey@0"
I20260620 23:26:43.397550 21222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "6b33d862e0f54733822d0ce15252fa62"
format_stamp: "Formatted at 2026-06-20 23:26:43 on dist-test-slave-5kzz"
server_key: "25842f04ab17c91d87487d420c55b462"
server_key_iv: "3cdb0707a28ddb5a082865cf0360962b"
server_key_version: "encryptionkey@0"
I20260620 23:26:43.403358 21222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:26:43.407613 21237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:43.409142 21222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:26:43.409368 21222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "6b33d862e0f54733822d0ce15252fa62"
format_stamp: "Formatted at 2026-06-20 23:26:43 on dist-test-slave-5kzz"
server_key: "25842f04ab17c91d87487d420c55b462"
server_key_iv: "3cdb0707a28ddb5a082865cf0360962b"
server_key_version: "encryptionkey@0"
I20260620 23:26:43.409593 21222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:43.428289 21222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:43.432929 21222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:43.433223 21222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:43.452962 21222 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41523
I20260620 23:26:43.453011 21288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41523 every 8 connection(s)
I20260620 23:26:43.454368 21222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:43.458403 21289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:43.461349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 21222
I20260620 23:26:43.461547 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:43.462069 388 external_mini_cluster.cc:1428] Setting key 0fae052e813de337ad625768267f9e48
I20260620 23:26:43.470908 21289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62: Bootstrap starting.
I20260620 23:26:43.474740 21289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:43.475863 21289 log.cc:824] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:43.476184 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:43.463984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59255 (local address 127.0.0.1:41523)
0620 23:26:43.464795 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:43.464806 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:43.464854 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:26:43.466531 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:43.466554 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:43.467047 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:43.467405 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:43.468075 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:43.469447 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:43.471376 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:43.472079 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:43.472217 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:43.473538 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:43.473546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:43.473560 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:43.474170 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:43.474249 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:43.474275 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:43.474377 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:43.474658 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:26:43.475102 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":374,"threads_started":1}
I20260620 23:26:43.478924 21289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62: No bootstrap required, opened a new log
I20260620 23:26:43.479803 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:43.489101 21289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b33d862e0f54733822d0ce15252fa62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41523 } }
I20260620 23:26:43.489629 21289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:43.489748 21289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b33d862e0f54733822d0ce15252fa62, State: Initialized, Role: FOLLOWER
I20260620 23:26:43.490420 21289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b33d862e0f54733822d0ce15252fa62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41523 } }
I20260620 23:26:43.490669 21289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:43.490885 21289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:43.491086 21289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:43.491777 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:43.484964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41095 (local address 127.0.0.1:41523)
0620 23:26:43.485154 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:43.485162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:43.485630 (+ 468us) server_negotiation.cc:407] Connection header received
0620 23:26:43.485786 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:43.485792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:43.485875 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:43.485977 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:43.486492 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:43.487680 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:43.489060 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:43.489720 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:43.489860 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:43.490959 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:43.490967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:43.490974 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:43.490996 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:43.491045 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:43.491070 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:43.491185 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:43.491328 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 23:26:43.491438 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:26:43.493650 21289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b33d862e0f54733822d0ce15252fa62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41523 } }
I20260620 23:26:43.494187 21289 leader_election.cc:302] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b33d862e0f54733822d0ce15252fa62; no voters:
I20260620 23:26:43.494819 21289 leader_election.cc:288] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:43.494975 21299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:43.495666 21299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [term 1 LEADER]: Becoming Leader. State: Replica: 6b33d862e0f54733822d0ce15252fa62, State: Running, Role: LEADER
I20260620 23:26:43.496163 21299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b33d862e0f54733822d0ce15252fa62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41523 } }
I20260620 23:26:43.497066 21289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:43.499153 21301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b33d862e0f54733822d0ce15252fa62. Latest consensus state: current_term: 1 leader_uuid: "6b33d862e0f54733822d0ce15252fa62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b33d862e0f54733822d0ce15252fa62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41523 } } }
I20260620 23:26:43.499528 21301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:43.499248 21300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b33d862e0f54733822d0ce15252fa62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b33d862e0f54733822d0ce15252fa62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41523 } } }
I20260620 23:26:43.499640 21300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:43.502449 21304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:43.507813 21304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:43.518275 21304 catalog_manager.cc:1347] Generated new cluster ID: 7f19988e0047494685217f4952eb5700
I20260620 23:26:43.518452 21304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:43.542606 21304 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:43.544090 21304 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:43.560298 21304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b33d862e0f54733822d0ce15252fa62: Generated new TSK 0
I20260620 23:26:43.561214 21304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:43.578931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21222
W20260620 23:26:43.580642 21284 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59255
W20260620 23:26:43.581457 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21222: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:26:43.581619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 21222
2026-06-20T23:26:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1953 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:43 dist-test-slave-5kzz krb5kdc[21320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:43 dist-test-slave-5kzz krb5kdc[21320](info): set up 2 sockets
Jun 20 23:26:43 dist-test-slave-5kzz krb5kdc[21320](info): commencing operation
krb5kdc: starting...
W20260620 23:26:44.715204 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:44 dist-test-slave-5kzz krb5kdc[21320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:26:44.974057 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:45.347254 21333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:45.347574 21333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:45.347747 21333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:45.355394 21333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:45.355523 21333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:45.355665 21333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:45.355767 21333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:45.355844 21333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:45.365885 21333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:45.367249 21333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:45.368577 21333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:45.381433 21338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:45.381389 21339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:45.381493 21333 server_base.cc:1034] running on GCE node
W20260620 23:26:45.381837 21341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:45.382822 21333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:45.384213 21333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:45.385424 21333 hybrid_clock.cc:648] HybridClock initialized: now 1781998005385361 us; error 52 us; skew 500 ppm
I20260620 23:26:45.385787 21333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:45 dist-test-slave-5kzz krb5kdc[21320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:45.389132 21333 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:26:45.390612 21333 webserver.cc:460] Webserver started at http://127.0.0.1:39995/ using document root <none> and password file <none>
I20260620 23:26:45.391352 21333 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:45.391482 21333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:45.391800 21333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:45.394986 21333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "9ed510023b5b4cf6b373231f1e439f5b"
format_stamp: "Formatted at 2026-06-20 23:26:45 on dist-test-slave-5kzz"
server_key: "6140e161b2654b1d6b0c2d0584196d61"
server_key_iv: "aca864b1b41a1569b0ed1a27af69e662"
server_key_version: "encryptionkey@0"
I20260620 23:26:45.395819 21333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "9ed510023b5b4cf6b373231f1e439f5b"
format_stamp: "Formatted at 2026-06-20 23:26:45 on dist-test-slave-5kzz"
server_key: "6140e161b2654b1d6b0c2d0584196d61"
server_key_iv: "aca864b1b41a1569b0ed1a27af69e662"
server_key_version: "encryptionkey@0"
I20260620 23:26:45.401319 21333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:26:45.405017 21348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:45.406545 21333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:26:45.406770 21333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "9ed510023b5b4cf6b373231f1e439f5b"
format_stamp: "Formatted at 2026-06-20 23:26:45 on dist-test-slave-5kzz"
server_key: "6140e161b2654b1d6b0c2d0584196d61"
server_key_iv: "aca864b1b41a1569b0ed1a27af69e662"
server_key_version: "encryptionkey@0"
I20260620 23:26:45.406986 21333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:45.413144 21333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:45.417671 21333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:45.418037 21333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:45.438609 21333 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45293
I20260620 23:26:45.438647 21399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45293 every 8 connection(s)
I20260620 23:26:45.440204 21333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:45.444001 21400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:45.450358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 21333
I20260620 23:26:45.450557 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:45.451123 388 external_mini_cluster.cc:1428] Setting key 4b6acb4b984f61374126072fae33474b
I20260620 23:26:45.455367 21400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b: Bootstrap starting.
I20260620 23:26:45.459077 21400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:45.459916 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:45.453189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52519 (local address 127.0.0.1:45293)
0620 23:26:45.454138 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:45.454150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:45.454205 (+ 55us) server_negotiation.cc:407] Connection header received
0620 23:26:45.456257 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:45.456282 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:45.456651 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:45.457047 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:45.458189 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:45.458196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:45.458210 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:45.458228 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:45.458303 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:45.458327 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:45.458481 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:45.458631 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 23:26:45.458853 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":455,"threads_started":1}
I20260620 23:26:45.460224 21400 log.cc:824] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:45.463788 21400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b: No bootstrap required, opened a new log
I20260620 23:26:45.464286 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:45.471251 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:45.468559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32837 (local address 127.0.0.1:45293)
0620 23:26:45.468757 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:45.468764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:45.469180 (+ 416us) server_negotiation.cc:407] Connection header received
0620 23:26:45.469321 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:45.469327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:45.469401 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:45.469524 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:45.470464 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:45.470470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:45.470475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:45.470494 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:45.470545 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:45.470570 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:45.470763 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:45.470891 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 23:26:45.470958 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:26:45.472461 21400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed510023b5b4cf6b373231f1e439f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45293 } }
I20260620 23:26:45.472888 21400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:45.472992 21400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed510023b5b4cf6b373231f1e439f5b, State: Initialized, Role: FOLLOWER
I20260620 23:26:45.473569 21400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed510023b5b4cf6b373231f1e439f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45293 } }
I20260620 23:26:45.473815 21400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:45.473954 21400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:45.474116 21400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:45.476372 21400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed510023b5b4cf6b373231f1e439f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45293 } }
I20260620 23:26:45.476884 21400 leader_election.cc:302] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ed510023b5b4cf6b373231f1e439f5b; no voters:
I20260620 23:26:45.477475 21400 leader_election.cc:288] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:45.477624 21410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:45.479156 21410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [term 1 LEADER]: Becoming Leader. State: Replica: 9ed510023b5b4cf6b373231f1e439f5b, State: Running, Role: LEADER
I20260620 23:26:45.479803 21400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:45.479779 21410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed510023b5b4cf6b373231f1e439f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45293 } }
I20260620 23:26:45.483448 21412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ed510023b5b4cf6b373231f1e439f5b. Latest consensus state: current_term: 1 leader_uuid: "9ed510023b5b4cf6b373231f1e439f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed510023b5b4cf6b373231f1e439f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45293 } } }
I20260620 23:26:45.484094 21412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:45.483542 21411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ed510023b5b4cf6b373231f1e439f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed510023b5b4cf6b373231f1e439f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45293 } } }
I20260620 23:26:45.484645 21411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:45.492239 21418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:45.496713 21418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:45.505662 21418 catalog_manager.cc:1347] Generated new cluster ID: b23feae332924917bb2e9c1491abb3dd
I20260620 23:26:45.505800 21418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:45.530961 21418 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:45.531443 21418 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:45.547257 21418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ed510023b5b4cf6b373231f1e439f5b: Generated new TSK 0
I20260620 23:26:45.548230 21418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:45.558034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21333
W20260620 23:26:45.559948 21395 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52519
W20260620 23:26:45.560724 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21333: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:26:45.560892 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 21333
2026-06-20T23:26:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1981 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:45 dist-test-slave-5kzz krb5kdc[21431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:45 dist-test-slave-5kzz krb5kdc[21431](info): set up 2 sockets
Jun 20 23:26:45 dist-test-slave-5kzz krb5kdc[21431](info): commencing operation
krb5kdc: starting...
W20260620 23:26:46.675256 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:46 dist-test-slave-5kzz krb5kdc[21431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:26:46.942144 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:47.282445 21444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:47.282904 21444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:47.283077 21444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:47.290436 21444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:47.290553 21444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:47.290704 21444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:47.290787 21444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:47.290843 21444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:47.300987 21444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:47.302371 21444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:47.303753 21444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:47.316090 21452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:47.316520 21449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:47.316555 21450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:47.317039 21444 server_base.cc:1034] running on GCE node
I20260620 23:26:47.317917 21444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:47.319537 21444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:47.320792 21444 hybrid_clock.cc:648] HybridClock initialized: now 1781998007320735 us; error 50 us; skew 500 ppm
I20260620 23:26:47.321177 21444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:47 dist-test-slave-5kzz krb5kdc[21431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:47.324689 21444 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:26:47.326491 21444 webserver.cc:460] Webserver started at http://127.0.0.1:39087/ using document root <none> and password file <none>
I20260620 23:26:47.327314 21444 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:47.327467 21444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:47.327778 21444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:47.330977 21444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "bd918fd321ae4753a3befb61a9e62c67"
format_stamp: "Formatted at 2026-06-20 23:26:47 on dist-test-slave-5kzz"
server_key: "6ebc7213a6dc3b2ba11b1217ff5aa0f5"
server_key_iv: "9285b78cf0678d8814bec8815c42412e"
server_key_version: "encryptionkey@0"
I20260620 23:26:47.331758 21444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "bd918fd321ae4753a3befb61a9e62c67"
format_stamp: "Formatted at 2026-06-20 23:26:47 on dist-test-slave-5kzz"
server_key: "6ebc7213a6dc3b2ba11b1217ff5aa0f5"
server_key_iv: "9285b78cf0678d8814bec8815c42412e"
server_key_version: "encryptionkey@0"
I20260620 23:26:47.337000 21444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:26:47.340631 21459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:47.342047 21444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:26:47.342267 21444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "bd918fd321ae4753a3befb61a9e62c67"
format_stamp: "Formatted at 2026-06-20 23:26:47 on dist-test-slave-5kzz"
server_key: "6ebc7213a6dc3b2ba11b1217ff5aa0f5"
server_key_iv: "9285b78cf0678d8814bec8815c42412e"
server_key_version: "encryptionkey@0"
I20260620 23:26:47.342535 21444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:47.379266 21444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:47.383780 21444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:47.384029 21444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:47.403002 21444 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33037
I20260620 23:26:47.403038 21510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33037 every 8 connection(s)
I20260620 23:26:47.404484 21444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:47.408360 21511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:47.408530 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 21444
I20260620 23:26:47.408716 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:47.409267 388 external_mini_cluster.cc:1428] Setting key 449658398cf611018b31383dd5708adf
I20260620 23:26:47.423729 21511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67: Bootstrap starting.
Jun 20 23:26:47 dist-test-slave-5kzz krb5kdc[21431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:26:47.427000 21511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:47.428086 21511 log.cc:824] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:47.430974 21511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67: No bootstrap required, opened a new log
I20260620 23:26:47.432971 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:47.411187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:42625 (local address 127.0.0.1:33037)
0620 23:26:47.412026 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:47.412038 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:47.412090 (+ 52us) server_negotiation.cc:407] Connection header received
0620 23:26:47.413702 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:47.413726 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:47.414161 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:47.414522 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:47.416327 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:47.417622 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:47.419487 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:47.420260 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:47.420407 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:47.425360 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:47.425369 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:47.425383 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:47.425426 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:47.428634 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:47.429516 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:47.429524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:47.429530 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:47.429711 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:47.430193 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:47.430197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:47.430199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:47.430821 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:47.431152 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:47.431635 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 23:26:47.431954 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":395,"threads_started":1}
I20260620 23:26:47.437448 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:47.439793 21511 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd918fd321ae4753a3befb61a9e62c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260620 23:26:47.440249 21511 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:47.440375 21511 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd918fd321ae4753a3befb61a9e62c67, State: Initialized, Role: FOLLOWER
I20260620 23:26:47.441032 21511 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd918fd321ae4753a3befb61a9e62c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260620 23:26:47.441282 21511 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:47.441445 21511 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:47.441614 21511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:47.445178 21511 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd918fd321ae4753a3befb61a9e62c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260620 23:26:47.445777 21511 leader_election.cc:302] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd918fd321ae4753a3befb61a9e62c67; no voters:
I20260620 23:26:47.446424 21511 leader_election.cc:288] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:47.446554 21521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:47.447346 21521 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [term 1 LEADER]: Becoming Leader. State: Replica: bd918fd321ae4753a3befb61a9e62c67, State: Running, Role: LEADER
I20260620 23:26:47.448012 21521 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd918fd321ae4753a3befb61a9e62c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260620 23:26:47.450088 21521 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd918fd321ae4753a3befb61a9e62c67. Latest consensus state: current_term: 1 leader_uuid: "bd918fd321ae4753a3befb61a9e62c67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd918fd321ae4753a3befb61a9e62c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } } }
I20260620 23:26:47.450452 21521 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:47.451268 21523 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd918fd321ae4753a3befb61a9e62c67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd918fd321ae4753a3befb61a9e62c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } } }
I20260620 23:26:47.454232 21523 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:47.455415 21511 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:47.457531 21525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:47.464608 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:47.442337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:51083 (local address 127.0.0.1:33037)
0620 23:26:47.442503 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:47.442508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:47.442901 (+ 393us) server_negotiation.cc:407] Connection header received
0620 23:26:47.443087 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:47.443090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:47.443161 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:47.443251 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:47.445079 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:47.446058 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:47.448436 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:47.449115 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:47.449262 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:47.454635 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:47.454645 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:47.454653 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:47.454733 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:47.459607 (+ 4874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:47.461055 (+ 1448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:47.461063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:47.461069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:47.461250 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:47.461989 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:47.461997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:47.462003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:47.462512 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:47.463810 (+ 1298us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:47.463992 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 23:26:47.464193 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:26:47.468268 21525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:47.480646 21525 catalog_manager.cc:1347] Generated new cluster ID: 47c6f1f104b041d89913561f801028d0
I20260620 23:26:47.480829 21525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:47.506621 21525 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:47.508308 21525 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:47.521268 21525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd918fd321ae4753a3befb61a9e62c67: Generated new TSK 0
I20260620 23:26:47.522212 21525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:47.537683 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21444
I20260620 23:26:47.539031 21506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:47.640960 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 21444
W20260620 23:26:47.658463 21429 connection.cc:535] client connection to 127.0.0.1:33037 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33037): Connection reset by peer (error 104)
2026-06-20T23:26:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:47 dist-test-slave-5kzz krb5kdc[21543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:47 dist-test-slave-5kzz krb5kdc[21543](info): set up 2 sockets
Jun 20 23:26:47 dist-test-slave-5kzz krb5kdc[21543](info): commencing operation
krb5kdc: starting...
W20260620 23:26:48.771310 388 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:48 dist-test-slave-5kzz krb5kdc[21543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:26:49.056123 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:49.397111 21556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:49.397425 21556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:49.397600 21556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:49.404956 21556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:49.405089 21556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:49.405185 21556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:49.405284 21556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:49.405371 21556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:49.415659 21556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:49.417004 21556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:49.418366 21556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:49.431340 21564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:49.431474 21561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:49.431371 21562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:49.432657 21556 server_base.cc:1034] running on GCE node
I20260620 23:26:49.433526 21556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:49.435107 21556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:49.436347 21556 hybrid_clock.cc:648] HybridClock initialized: now 1781998009436282 us; error 59 us; skew 500 ppm
I20260620 23:26:49.436723 21556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:49 dist-test-slave-5kzz krb5kdc[21543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:49.443686 21556 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:26:49.445277 21556 webserver.cc:460] Webserver started at http://127.0.0.1:45743/ using document root <none> and password file <none>
I20260620 23:26:49.446000 21556 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:49.446139 21556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:49.446462 21556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:49.449625 21556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "cfbeb4f95f8f4faea989c86ad28a017e"
format_stamp: "Formatted at 2026-06-20 23:26:49 on dist-test-slave-5kzz"
server_key: "0f87322c5136747ba26fd38bb37fca85"
server_key_iv: "6c6987832941f06752e943e8cd0d78ac"
server_key_version: "encryptionkey@0"
I20260620 23:26:49.450424 21556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "cfbeb4f95f8f4faea989c86ad28a017e"
format_stamp: "Formatted at 2026-06-20 23:26:49 on dist-test-slave-5kzz"
server_key: "0f87322c5136747ba26fd38bb37fca85"
server_key_iv: "6c6987832941f06752e943e8cd0d78ac"
server_key_version: "encryptionkey@0"
I20260620 23:26:49.455844 21556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:26:49.459372 21571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:49.460805 21556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:26:49.460986 21556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "cfbeb4f95f8f4faea989c86ad28a017e"
format_stamp: "Formatted at 2026-06-20 23:26:49 on dist-test-slave-5kzz"
server_key: "0f87322c5136747ba26fd38bb37fca85"
server_key_iv: "6c6987832941f06752e943e8cd0d78ac"
server_key_version: "encryptionkey@0"
I20260620 23:26:49.461199 21556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:49.487200 21556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:49.491643 21556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:49.491972 21556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:49.511943 21556 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33575
I20260620 23:26:49.511997 21622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33575 every 8 connection(s)
I20260620 23:26:49.513432 21556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:49.517346 21623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:49.523296 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 21556
I20260620 23:26:49.523526 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:49.523972 388 external_mini_cluster.cc:1428] Setting key 25ad18067b1c5e518845f9a19955e0af
I20260620 23:26:49.528893 21623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e: Bootstrap starting.
I20260620 23:26:49.532663 21623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:49.533987 21623 log.cc:824] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:49.537436 21623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e: No bootstrap required, opened a new log
I20260620 23:26:49.538932 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:49.526146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:45605 (local address 127.0.0.1:33575)
0620 23:26:49.527081 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:49.527092 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:49.527135 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:26:49.528860 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:49.528886 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:49.529372 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:49.529711 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:49.530365 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:49.531812 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:49.533826 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:49.534978 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:49.535197 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:49.536412 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:49.536419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:49.536434 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:49.536986 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:49.537061 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:49.537097 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:49.537208 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:49.537740 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 23:26:49.537882 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":464,"threads_started":1}
I20260620 23:26:49.542840 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:49.545748 21623 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfbeb4f95f8f4faea989c86ad28a017e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33575 } }
I20260620 23:26:49.546160 21623 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:49.546289 21623 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfbeb4f95f8f4faea989c86ad28a017e, State: Initialized, Role: FOLLOWER
I20260620 23:26:49.546936 21623 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfbeb4f95f8f4faea989c86ad28a017e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33575 } }
I20260620 23:26:49.547194 21623 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:49.547353 21623 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:49.547626 21623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:49.550967 21623 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfbeb4f95f8f4faea989c86ad28a017e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33575 } }
I20260620 23:26:49.551652 21623 leader_election.cc:302] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfbeb4f95f8f4faea989c86ad28a017e; no voters:
I20260620 23:26:49.552438 21623 leader_election.cc:288] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:49.552605 21633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:49.554123 21633 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [term 1 LEADER]: Becoming Leader. State: Replica: cfbeb4f95f8f4faea989c86ad28a017e, State: Running, Role: LEADER
I20260620 23:26:49.555037 21633 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfbeb4f95f8f4faea989c86ad28a017e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33575 } }
I20260620 23:26:49.555935 21623 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:49.556062 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:49.547344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:36985 (local address 127.0.0.1:33575)
0620 23:26:49.547541 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:49.547548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:49.548080 (+ 532us) server_negotiation.cc:407] Connection header received
0620 23:26:49.548265 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:49.548268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:49.548333 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:49.548423 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:49.549048 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:49.549843 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:49.552013 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:49.552998 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:49.553194 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:49.554858 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:49.554864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:49.554869 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:49.554906 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:49.554981 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:49.555006 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:49.555103 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:49.555647 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 23:26:49.555714 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:26:49.564450 21634 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfbeb4f95f8f4faea989c86ad28a017e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfbeb4f95f8f4faea989c86ad28a017e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33575 } } }
I20260620 23:26:49.564983 21634 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:49.566171 21635 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfbeb4f95f8f4faea989c86ad28a017e. Latest consensus state: current_term: 1 leader_uuid: "cfbeb4f95f8f4faea989c86ad28a017e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfbeb4f95f8f4faea989c86ad28a017e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33575 } } }
I20260620 23:26:49.566509 21635 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:49.567875 21642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:49.574091 21642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:49.585507 21642 catalog_manager.cc:1347] Generated new cluster ID: 3f0a4c7c5496436090e455fd6218600e
I20260620 23:26:49.585652 21642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:49.625252 21642 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:49.626928 21642 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:49.638545 21642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfbeb4f95f8f4faea989c86ad28a017e: Generated new TSK 0
I20260620 23:26:49.639755 21642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:49.657678 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21556
W20260620 23:26:49.659124 21618 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.1.12:45605
W20260620 23:26:49.659958 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21556: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:26:49.660102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 21556
W20260620 23:26:49.678403 21541 connection.cc:535] client connection to 127.0.0.1:33575 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33575): Connection reset by peer (error 104)
2026-06-20T23:26:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2020 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:49 dist-test-slave-5kzz krb5kdc[21654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:49 dist-test-slave-5kzz krb5kdc[21654](info): set up 2 sockets
Jun 20 23:26:49 dist-test-slave-5kzz krb5kdc[21654](info): commencing operation
krb5kdc: starting...
W20260620 23:26:50.783315 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:50 dist-test-slave-5kzz krb5kdc[21654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:26:51.071277 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:51.423067 21667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:51.423473 21667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:51.423650 21667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:51.431085 21667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:51.431272 21667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:51.431389 21667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:51.431509 21667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:51.431593 21667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:51.441884 21667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:51.443320 21667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:51.444698 21667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:51.459007 21672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:51.459059 21675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:51.459035 21673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:51.460057 21667 server_base.cc:1034] running on GCE node
I20260620 23:26:51.460903 21667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:51.462590 21667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:51.463904 21667 hybrid_clock.cc:648] HybridClock initialized: now 1781998011463856 us; error 47 us; skew 500 ppm
I20260620 23:26:51.464275 21667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:51 dist-test-slave-5kzz krb5kdc[21654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:51.467577 21667 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:26:51.469271 21667 webserver.cc:460] Webserver started at http://127.0.0.1:38435/ using document root <none> and password file <none>
I20260620 23:26:51.470116 21667 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:51.470276 21667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:51.470618 21667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:51.473774 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "1376ed90b8ef402d99e8928e04940ade"
format_stamp: "Formatted at 2026-06-20 23:26:51 on dist-test-slave-5kzz"
server_key: "df8bfc71582cb3335042497b5b9a8d66"
server_key_iv: "54966cbe28f5d5004a2949567ba34ec4"
server_key_version: "encryptionkey@0"
I20260620 23:26:51.474601 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "1376ed90b8ef402d99e8928e04940ade"
format_stamp: "Formatted at 2026-06-20 23:26:51 on dist-test-slave-5kzz"
server_key: "df8bfc71582cb3335042497b5b9a8d66"
server_key_iv: "54966cbe28f5d5004a2949567ba34ec4"
server_key_version: "encryptionkey@0"
I20260620 23:26:51.480175 21667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 23:26:51.483863 21682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:51.485242 21667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:26:51.485440 21667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "1376ed90b8ef402d99e8928e04940ade"
format_stamp: "Formatted at 2026-06-20 23:26:51 on dist-test-slave-5kzz"
server_key: "df8bfc71582cb3335042497b5b9a8d66"
server_key_iv: "54966cbe28f5d5004a2949567ba34ec4"
server_key_version: "encryptionkey@0"
I20260620 23:26:51.485661 21667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:51.491923 21667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:51.496482 21667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:51.496863 21667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:51.518102 21667 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38161
I20260620 23:26:51.518162 21733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38161 every 8 connection(s)
I20260620 23:26:51.519694 21667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:51.523789 21734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:51.528246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 21667
I20260620 23:26:51.528467 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:51.528890 388 external_mini_cluster.cc:1428] Setting key f5a1d65b720699197a68635171b0a74c
I20260620 23:26:51.536926 21734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade: Bootstrap starting.
I20260620 23:26:51.539139 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:51.530904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:56627 (local address 127.0.0.1:38161)
0620 23:26:51.531732 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:51.531750 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:51.531814 (+ 64us) server_negotiation.cc:407] Connection header received
0620 23:26:51.534949 (+ 3135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:51.534984 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:51.535506 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:51.536059 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:51.537077 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:51.537086 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:51.537117 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:51.537134 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:51.537206 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:51.537241 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:51.537456 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:51.537588 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 23:26:51.537802 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":343,"threads_started":1}
I20260620 23:26:51.540858 21734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:51.542263 21734 log.cc:824] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:51.543721 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:51.545404 21734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade: No bootstrap required, opened a new log
I20260620 23:26:51.551548 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:51.548259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:47337 (local address 127.0.0.1:38161)
0620 23:26:51.548463 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:51.548470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:51.548974 (+ 504us) server_negotiation.cc:407] Connection header received
0620 23:26:51.549104 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:51.549110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:51.549176 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:51.549295 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:51.550812 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:51.550817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:51.550822 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:51.550842 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:51.550896 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:51.550917 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:51.551048 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:51.551185 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:26:51.551252 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:26:51.554199 21734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1376ed90b8ef402d99e8928e04940ade" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } }
I20260620 23:26:51.554646 21734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:51.554803 21734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1376ed90b8ef402d99e8928e04940ade, State: Initialized, Role: FOLLOWER
I20260620 23:26:51.555454 21734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1376ed90b8ef402d99e8928e04940ade" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } }
I20260620 23:26:51.555706 21734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:51.555864 21734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:51.556052 21734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:51.558400 21734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1376ed90b8ef402d99e8928e04940ade" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } }
I20260620 23:26:51.559051 21734 leader_election.cc:302] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1376ed90b8ef402d99e8928e04940ade; no voters:
I20260620 23:26:51.559638 21734 leader_election.cc:288] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:51.559809 21744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:51.560472 21744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [term 1 LEADER]: Becoming Leader. State: Replica: 1376ed90b8ef402d99e8928e04940ade, State: Running, Role: LEADER
I20260620 23:26:51.561074 21744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1376ed90b8ef402d99e8928e04940ade" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } }
I20260620 23:26:51.561903 21734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:51.564188 21746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1376ed90b8ef402d99e8928e04940ade. Latest consensus state: current_term: 1 leader_uuid: "1376ed90b8ef402d99e8928e04940ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1376ed90b8ef402d99e8928e04940ade" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } } }
I20260620 23:26:51.564571 21746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:51.564301 21745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1376ed90b8ef402d99e8928e04940ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1376ed90b8ef402d99e8928e04940ade" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } } }
I20260620 23:26:51.564718 21745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:51.565584 21751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:51.571992 21751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:51.583678 21751 catalog_manager.cc:1347] Generated new cluster ID: 5cad5cc20b69496f9820efd31b912dcc
I20260620 23:26:51.583809 21751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:51.617158 21751 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:51.617645 21751 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:51.624800 21751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1376ed90b8ef402d99e8928e04940ade: Generated new TSK 0
I20260620 23:26:51.625638 21751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:51.637254 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21667
W20260620 23:26:51.638447 21729 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.1.12:56627
W20260620 23:26:51.639328 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21667: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:26:51.639516 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 21667
2026-06-20T23:26:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1977 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:51 dist-test-slave-5kzz krb5kdc[21765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:51 dist-test-slave-5kzz krb5kdc[21765](info): set up 2 sockets
Jun 20 23:26:51 dist-test-slave-5kzz krb5kdc[21765](info): commencing operation
krb5kdc: starting...
W20260620 23:26:52.763320 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:52 dist-test-slave-5kzz krb5kdc[21765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:26:53.048161 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:53.392217 21778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:53.392593 21778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:53.392755 21778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:53.400087 21778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:53.400203 21778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:53.400249 21778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:53.400309 21778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:53.400352 21778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:53.410153 21778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:53.411403 21778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:53.412674 21778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:53.424265 21784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:53.424352 21786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:53.424309 21783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:53.425295 21778 server_base.cc:1034] running on GCE node
I20260620 23:26:53.426177 21778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:53.427613 21778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:53.428804 21778 hybrid_clock.cc:648] HybridClock initialized: now 1781998013428758 us; error 46 us; skew 500 ppm
I20260620 23:26:53.429150 21778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:53 dist-test-slave-5kzz krb5kdc[21765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:53.432336 21778 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:26:53.433856 21778 webserver.cc:460] Webserver started at http://127.0.0.1:41797/ using document root <none> and password file <none>
I20260620 23:26:53.434552 21778 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:53.434747 21778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:53.435062 21778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:53.438041 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "76d8495783774fcba58b2a61c77b8846"
format_stamp: "Formatted at 2026-06-20 23:26:53 on dist-test-slave-5kzz"
server_key: "5bb6617db3bf0f8449beaef9d04bd3e8"
server_key_iv: "5653a17870ff11daed499e2b8d829a8d"
server_key_version: "encryptionkey@0"
I20260620 23:26:53.438889 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "76d8495783774fcba58b2a61c77b8846"
format_stamp: "Formatted at 2026-06-20 23:26:53 on dist-test-slave-5kzz"
server_key: "5bb6617db3bf0f8449beaef9d04bd3e8"
server_key_iv: "5653a17870ff11daed499e2b8d829a8d"
server_key_version: "encryptionkey@0"
I20260620 23:26:53.444133 21778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:26:53.447786 21793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:53.449209 21778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:26:53.449395 21778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "76d8495783774fcba58b2a61c77b8846"
format_stamp: "Formatted at 2026-06-20 23:26:53 on dist-test-slave-5kzz"
server_key: "5bb6617db3bf0f8449beaef9d04bd3e8"
server_key_iv: "5653a17870ff11daed499e2b8d829a8d"
server_key_version: "encryptionkey@0"
I20260620 23:26:53.449609 21778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:53.463573 21778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:53.467968 21778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:53.468264 21778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:53.486168 21778 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45213
I20260620 23:26:53.486199 21844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45213 every 8 connection(s)
I20260620 23:26:53.487651 21778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:53.491506 21845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:53.495103 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 21778
I20260620 23:26:53.495285 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:53.495790 388 external_mini_cluster.cc:1428] Setting key 719c4b57999525ae639484d3fa61f9c2
I20260620 23:26:53.503685 21845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846: Bootstrap starting.
I20260620 23:26:53.506791 21845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:53.508157 21845 log.cc:824] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:26:53 dist-test-slave-5kzz krb5kdc[21765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:26:53.511149 21845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846: No bootstrap required, opened a new log
I20260620 23:26:53.518951 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:53.497663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:59049 (local address 127.0.0.1:45213)
0620 23:26:53.498381 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:53.498394 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:53.498433 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:26:53.500189 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:53.500212 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:53.500649 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:53.501057 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:53.502543 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:53.503954 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:53.505203 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:53.505877 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:53.506069 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:53.510888 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:53.510897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:53.510912 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:53.510954 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:53.514175 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:53.515096 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:53.515103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:53.515110 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:53.515289 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:53.515782 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:53.515786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:53.515788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:53.516515 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:53.516925 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:53.517426 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 23:26:53.517782 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":301,"threads_started":1}
I20260620 23:26:53.519538 21845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d8495783774fcba58b2a61c77b8846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45213 } }
I20260620 23:26:53.519951 21845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:53.520066 21845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d8495783774fcba58b2a61c77b8846, State: Initialized, Role: FOLLOWER
I20260620 23:26:53.520723 21845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d8495783774fcba58b2a61c77b8846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45213 } }
I20260620 23:26:53.520973 21845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:53.521136 21845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:53.521306 21845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:53.523217 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:53.524080 21845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d8495783774fcba58b2a61c77b8846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45213 } }
I20260620 23:26:53.524606 21845 leader_election.cc:302] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76d8495783774fcba58b2a61c77b8846; no voters:
I20260620 23:26:53.525230 21845 leader_election.cc:288] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:53.525394 21850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:53.526095 21850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [term 1 LEADER]: Becoming Leader. State: Replica: 76d8495783774fcba58b2a61c77b8846, State: Running, Role: LEADER
I20260620 23:26:53.526939 21850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d8495783774fcba58b2a61c77b8846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45213 } }
I20260620 23:26:53.529405 21850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d8495783774fcba58b2a61c77b8846. Latest consensus state: current_term: 1 leader_uuid: "76d8495783774fcba58b2a61c77b8846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d8495783774fcba58b2a61c77b8846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45213 } } }
I20260620 23:26:53.529837 21850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:53.530352 21852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76d8495783774fcba58b2a61c77b8846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d8495783774fcba58b2a61c77b8846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45213 } } }
I20260620 23:26:53.530823 21852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:53.540597 21845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:53.543937 21855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:53.551168 21855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:53.559806 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:53.541177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:53951 (local address 127.0.0.1:45213)
0620 23:26:53.541357 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:53.541364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:53.541902 (+ 538us) server_negotiation.cc:407] Connection header received
0620 23:26:53.542116 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:53.542121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:53.542221 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:53.542339 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:53.544476 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:53.545295 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:53.546765 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:53.547425 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:53.547564 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:53.551083 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:53.551092 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:53.551098 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:53.551139 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:53.555579 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:53.556648 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:53.556653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:53.556657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:53.556809 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:53.557395 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:53.557401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:53.557405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:53.557836 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:53.558127 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:53.559266 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0620 23:26:53.559460 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:26:53.565193 21855 catalog_manager.cc:1347] Generated new cluster ID: 1a64338c52a84fd7a7c49c80bb0d98aa
I20260620 23:26:53.565359 21855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:53.597911 21855 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:53.599308 21855 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:53.606817 21855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76d8495783774fcba58b2a61c77b8846: Generated new TSK 0
I20260620 23:26:53.607690 21855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:53.613341 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21778
I20260620 23:26:53.614514 21840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:53.705946 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 21778
W20260620 23:26:53.722898 21763 connection.cc:535] client connection to 127.0.0.1:45213 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45213): Connection reset by peer (error 104)
2026-06-20T23:26:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2065 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:53 dist-test-slave-5kzz krb5kdc[21877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:53 dist-test-slave-5kzz krb5kdc[21877](info): set up 2 sockets
Jun 20 23:26:53 dist-test-slave-5kzz krb5kdc[21877](info): commencing operation
krb5kdc: starting...
W20260620 23:26:54.823793 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:54 dist-test-slave-5kzz krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:26:55.073812 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:55.406956 21890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:55.407343 21890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:55.407516 21890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:55.414950 21890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:55.415123 21890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:55.415228 21890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:55.415339 21890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:55.415414 21890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:55.425680 21890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41997
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:55.427160 21890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:55.428505 21890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:55.440595 21895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:55.440485 21896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:55.440519 21898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:55.441648 21890 server_base.cc:1034] running on GCE node
I20260620 23:26:55.442466 21890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:55.444029 21890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:55.445268 21890 hybrid_clock.cc:648] HybridClock initialized: now 1781998015445209 us; error 51 us; skew 500 ppm
I20260620 23:26:55.445626 21890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:55 dist-test-slave-5kzz krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:55.454782 21890 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:26:55.456481 21890 webserver.cc:460] Webserver started at http://127.0.0.1:42475/ using document root <none> and password file <none>
I20260620 23:26:55.457233 21890 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:55.457376 21890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:55.457679 21890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:55.460826 21890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "91a59e4ab25b4baead00049efaa9eced"
format_stamp: "Formatted at 2026-06-20 23:26:55 on dist-test-slave-5kzz"
server_key: "3896a5d4da612ab27a68ccbf7b528ea0"
server_key_iv: "de1bc4cd5e08027172f19c9870555bb8"
server_key_version: "encryptionkey@0"
I20260620 23:26:55.461618 21890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "91a59e4ab25b4baead00049efaa9eced"
format_stamp: "Formatted at 2026-06-20 23:26:55 on dist-test-slave-5kzz"
server_key: "3896a5d4da612ab27a68ccbf7b528ea0"
server_key_iv: "de1bc4cd5e08027172f19c9870555bb8"
server_key_version: "encryptionkey@0"
I20260620 23:26:55.467623 21890 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 23:26:55.471510 21905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:55.472919 21890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:26:55.473104 21890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "91a59e4ab25b4baead00049efaa9eced"
format_stamp: "Formatted at 2026-06-20 23:26:55 on dist-test-slave-5kzz"
server_key: "3896a5d4da612ab27a68ccbf7b528ea0"
server_key_iv: "de1bc4cd5e08027172f19c9870555bb8"
server_key_version: "encryptionkey@0"
I20260620 23:26:55.473299 21890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:55.489322 21890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:55.493635 21890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:55.493937 21890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:55.512912 21890 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41997
I20260620 23:26:55.512931 21956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41997 every 8 connection(s)
I20260620 23:26:55.514324 21890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:55.518317 21957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:55.519517 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 21890
I20260620 23:26:55.519723 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:55.520257 388 external_mini_cluster.cc:1428] Setting key 12bc8ffef04b00985042e6955178a48a
I20260620 23:26:55.531538 21957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced: Bootstrap starting.
I20260620 23:26:55.533874 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:55.522232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:45509 (local address 127.0.0.1:41997)
0620 23:26:55.523094 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:55.523106 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:55.523159 (+ 53us) server_negotiation.cc:407] Connection header received
0620 23:26:55.525009 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:55.525036 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:55.525469 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:55.525882 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:55.526424 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:55.527727 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:55.529116 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:55.529789 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:55.529991 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:55.531529 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:55.531543 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:55.531562 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:55.532209 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:55.532291 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:55.532318 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:55.532490 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:55.532752 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 23:26:55.532939 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":388,"threads_started":1}
I20260620 23:26:55.534950 21957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:55.536017 21957 log.cc:824] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:55.537606 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:55.538923 21957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced: No bootstrap required, opened a new log
I20260620 23:26:55.549793 21957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a59e4ab25b4baead00049efaa9eced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41997 } }
I20260620 23:26:55.550204 21957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:55.550331 21957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a59e4ab25b4baead00049efaa9eced, State: Initialized, Role: FOLLOWER
I20260620 23:26:55.551007 21957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a59e4ab25b4baead00049efaa9eced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41997 } }
I20260620 23:26:55.551271 21957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:55.551421 21957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:55.551589 21957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:55.553284 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:55.541960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:58943 (local address 127.0.0.1:41997)
0620 23:26:55.542132 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:55.542136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:55.547252 (+ 5116us) server_negotiation.cc:407] Connection header received
0620 23:26:55.547476 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:55.547480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:55.547549 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:55.547674 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:55.548169 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:55.548982 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:55.550457 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:55.551038 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:55.551172 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:55.552494 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:55.552502 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:55.552505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:55.552527 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:55.552583 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:55.552599 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:55.552739 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:55.552947 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 23:26:55.553010 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:26:55.554296 21957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a59e4ab25b4baead00049efaa9eced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41997 } }
I20260620 23:26:55.554919 21957 leader_election.cc:302] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91a59e4ab25b4baead00049efaa9eced; no voters:
I20260620 23:26:55.555548 21957 leader_election.cc:288] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:55.555681 21967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:55.556447 21967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [term 1 LEADER]: Becoming Leader. State: Replica: 91a59e4ab25b4baead00049efaa9eced, State: Running, Role: LEADER
I20260620 23:26:55.557039 21967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a59e4ab25b4baead00049efaa9eced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41997 } }
I20260620 23:26:55.557744 21957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:55.559928 21969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91a59e4ab25b4baead00049efaa9eced. Latest consensus state: current_term: 1 leader_uuid: "91a59e4ab25b4baead00049efaa9eced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a59e4ab25b4baead00049efaa9eced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41997 } } }
I20260620 23:26:55.560417 21969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:55.562919 21968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91a59e4ab25b4baead00049efaa9eced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a59e4ab25b4baead00049efaa9eced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41997 } } }
I20260620 23:26:55.563208 21968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:55.563799 21974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:55.568198 21974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:55.581009 21974 catalog_manager.cc:1347] Generated new cluster ID: b259b5ab1a5f4c949e04e5147d68a556
I20260620 23:26:55.581137 21974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:55.599303 21974 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:55.600680 21974 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:55.606765 21974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91a59e4ab25b4baead00049efaa9eced: Generated new TSK 0
I20260620 23:26:55.607563 21974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:55.613441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21890
W20260620 23:26:55.614848 21952 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.1.12:45509
W20260620 23:26:55.615780 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 21890: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:26:55.615931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 21890
W20260620 23:26:55.633028 21875 connection.cc:535] client connection to 127.0.0.1:41997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41997): Connection reset by peer (error 104)
2026-06-20T23:26:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:55 dist-test-slave-5kzz krb5kdc[21988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:55 dist-test-slave-5kzz krb5kdc[21988](info): set up 2 sockets
Jun 20 23:26:55 dist-test-slave-5kzz krb5kdc[21988](info): commencing operation
krb5kdc: starting...
W20260620 23:26:56.723145 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:56 dist-test-slave-5kzz krb5kdc[21988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:26:56.975838 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:57.296989 22001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:57.297326 22001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:57.297500 22001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:57.304526 22001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:57.304672 22001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:57.304786 22001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:57.304898 22001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:57.304966 22001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:57.314635 22001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:57.315847 22001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:57.317046 22001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:57.328792 22006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:57.328851 22009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:57.328881 22007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:57.329869 22001 server_base.cc:1034] running on GCE node
I20260620 23:26:57.330629 22001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:57.331998 22001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:57.333195 22001 hybrid_clock.cc:648] HybridClock initialized: now 1781998017333142 us; error 53 us; skew 500 ppm
I20260620 23:26:57.333629 22001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:57 dist-test-slave-5kzz krb5kdc[21988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:57.336620 22001 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:26:57.337951 22001 webserver.cc:460] Webserver started at http://127.0.0.1:43181/ using document root <none> and password file <none>
I20260620 23:26:57.338613 22001 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:57.338769 22001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:57.339143 22001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:57.342036 22001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "f6f5147453aa405592a0dcab97ceeca4"
format_stamp: "Formatted at 2026-06-20 23:26:57 on dist-test-slave-5kzz"
server_key: "2236c1b9403fa4b355b41baad4494d41"
server_key_iv: "b2d15822b5b17b3a2cb1115b5ea4ebfe"
server_key_version: "encryptionkey@0"
I20260620 23:26:57.342903 22001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "f6f5147453aa405592a0dcab97ceeca4"
format_stamp: "Formatted at 2026-06-20 23:26:57 on dist-test-slave-5kzz"
server_key: "2236c1b9403fa4b355b41baad4494d41"
server_key_iv: "b2d15822b5b17b3a2cb1115b5ea4ebfe"
server_key_version: "encryptionkey@0"
I20260620 23:26:57.347873 22001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:26:57.351306 22016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:57.352578 22001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:26:57.352737 22001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "f6f5147453aa405592a0dcab97ceeca4"
format_stamp: "Formatted at 2026-06-20 23:26:57 on dist-test-slave-5kzz"
server_key: "2236c1b9403fa4b355b41baad4494d41"
server_key_iv: "b2d15822b5b17b3a2cb1115b5ea4ebfe"
server_key_version: "encryptionkey@0"
I20260620 23:26:57.352946 22001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:57.358331 22001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:57.362206 22001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:57.362491 22001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:57.380035 22001 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36465
I20260620 23:26:57.380119 22067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36465 every 8 connection(s)
I20260620 23:26:57.381407 22001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:57.385097 22068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:57.390173 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 22001
I20260620 23:26:57.390398 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:57.390930 388 external_mini_cluster.cc:1428] Setting key 081ceb936a158e997f9e3180fe63676b
I20260620 23:26:57.397640 22068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4: Bootstrap starting.
I20260620 23:26:57.400004 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:57.392875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:37275 (local address 127.0.0.1:36465)
0620 23:26:57.393657 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:57.393677 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:57.393740 (+ 63us) server_negotiation.cc:407] Connection header received
0620 23:26:57.396192 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:57.396221 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:57.396653 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:57.397084 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:57.398063 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:57.398071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:57.398087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:57.398103 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:57.398183 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:57.398210 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:57.398409 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:57.398533 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 23:26:57.398745 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":298,"threads_started":1}
I20260620 23:26:57.401343 22068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:57.402449 22068 log.cc:824] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:57.403610 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:57.405644 22068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4: No bootstrap required, opened a new log
I20260620 23:26:57.410938 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:57.408626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:49107 (local address 127.0.0.1:36465)
0620 23:26:57.408811 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:57.408817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:57.409135 (+ 318us) server_negotiation.cc:407] Connection header received
0620 23:26:57.409261 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:57.409266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:57.409326 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:57.409422 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:57.410274 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:57.410279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:57.410282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:26:57.410300 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:57.410344 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:26:57.410362 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:57.410484 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:57.410535 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 23:26:57.410594 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:26:57.414422 22068 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f5147453aa405592a0dcab97ceeca4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260620 23:26:57.414846 22068 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:57.414927 22068 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f5147453aa405592a0dcab97ceeca4, State: Initialized, Role: FOLLOWER
I20260620 23:26:57.415477 22068 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f5147453aa405592a0dcab97ceeca4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260620 23:26:57.415684 22068 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:57.415782 22068 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:57.415906 22068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:57.418035 22068 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f5147453aa405592a0dcab97ceeca4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260620 23:26:57.418470 22068 leader_election.cc:302] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f5147453aa405592a0dcab97ceeca4; no voters:
I20260620 23:26:57.419026 22068 leader_election.cc:288] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:57.419165 22078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:57.419852 22078 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [term 1 LEADER]: Becoming Leader. State: Replica: f6f5147453aa405592a0dcab97ceeca4, State: Running, Role: LEADER
I20260620 23:26:57.420338 22078 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f5147453aa405592a0dcab97ceeca4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260620 23:26:57.421257 22068 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:57.423169 22079 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6f5147453aa405592a0dcab97ceeca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f5147453aa405592a0dcab97ceeca4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } } }
I20260620 23:26:57.423487 22079 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:57.425590 22084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:57.425405 22080 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f5147453aa405592a0dcab97ceeca4. Latest consensus state: current_term: 1 leader_uuid: "f6f5147453aa405592a0dcab97ceeca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f5147453aa405592a0dcab97ceeca4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } } }
I20260620 23:26:57.425734 22080 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:57.431291 22084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:57.440047 22084 catalog_manager.cc:1347] Generated new cluster ID: a97d7a80a42a477c8ea384b44d1c2362
I20260620 23:26:57.440212 22084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:57.467767 22084 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:57.468237 22084 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:57.484141 22084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6f5147453aa405592a0dcab97ceeca4: Generated new TSK 0
I20260620 23:26:57.484977 22084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:57.500384 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22001
W20260620 23:26:57.501801 22063 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.1.12:37275
W20260620 23:26:57.502542 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22001: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:26:57.502741 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 22001
2026-06-20T23:26:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1886 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:57 dist-test-slave-5kzz krb5kdc[22099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:57 dist-test-slave-5kzz krb5kdc[22099](info): set up 2 sockets
Jun 20 23:26:57 dist-test-slave-5kzz krb5kdc[22099](info): commencing operation
krb5kdc: starting...
W20260620 23:26:58.610932 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:26:58 dist-test-slave-5kzz krb5kdc[22099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:26:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:26:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:26:58.853319 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41247
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:26:59.172596 22112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:26:59.172883 22112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:26:59.173043 22112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:26:59.180236 22112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:26:59.180346 22112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:26:59.180450 22112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:26:59.180552 22112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:26:59.180613 22112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:26:59.191035 22112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41247
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:26:59.192382 22112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:26:59.193703 22112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:26:59.205154 22117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:59.205201 22120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:26:59.205118 22118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:26:59.206828 22112 server_base.cc:1034] running on GCE node
I20260620 23:26:59.207679 22112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:26:59.209236 22112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:26:59.210490 22112 hybrid_clock.cc:648] HybridClock initialized: now 1781998019210431 us; error 44 us; skew 500 ppm
I20260620 23:26:59.210943 22112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:26:59 dist-test-slave-5kzz krb5kdc[22099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:26:59.214234 22112 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:26:59.215713 22112 webserver.cc:460] Webserver started at http://127.0.97.62:33133/ using document root <none> and password file <none>
I20260620 23:26:59.216374 22112 fs_manager.cc:359] Metadata directory not provided
I20260620 23:26:59.216475 22112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:26:59.216723 22112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:26:59.219655 22112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "afd91f546d78469a86493a8e3d0ac217"
format_stamp: "Formatted at 2026-06-20 23:26:59 on dist-test-slave-5kzz"
server_key: "13955ea441a3cb188edd47d0a7685714"
server_key_iv: "8d272a4e75f8a0b59af22912a599ab8f"
server_key_version: "encryptionkey@0"
I20260620 23:26:59.220381 22112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "afd91f546d78469a86493a8e3d0ac217"
format_stamp: "Formatted at 2026-06-20 23:26:59 on dist-test-slave-5kzz"
server_key: "13955ea441a3cb188edd47d0a7685714"
server_key_iv: "8d272a4e75f8a0b59af22912a599ab8f"
server_key_version: "encryptionkey@0"
I20260620 23:26:59.225400 22112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:26:59.229027 22127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:26:59.230428 22112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:26:59.230614 22112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "afd91f546d78469a86493a8e3d0ac217"
format_stamp: "Formatted at 2026-06-20 23:26:59 on dist-test-slave-5kzz"
server_key: "13955ea441a3cb188edd47d0a7685714"
server_key_iv: "8d272a4e75f8a0b59af22912a599ab8f"
server_key_version: "encryptionkey@0"
I20260620 23:26:59.230868 22112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:26:59.258121 22112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:26:59.262209 22112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:26:59.262512 22112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:26:59.281097 22112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41247
I20260620 23:26:59.281189 22178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41247 every 8 connection(s)
I20260620 23:26:59.282440 22112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:26:59.286062 22179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:26:59.286412 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 22112
I20260620 23:26:59.286581 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:26:59.287052 388 external_mini_cluster.cc:1428] Setting key 39bf748e6b89e132a4f76dfa8d427d3e
I20260620 23:26:59.299758 22179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217: Bootstrap starting.
Jun 20 23:26:59 dist-test-slave-5kzz krb5kdc[22099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:26:59.302882 22179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217: Neither blocks nor log segments found. Creating new log.
I20260620 23:26:59.304184 22179 log.cc:824] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217: Log is configured to *not* fsync() on all Append() calls
I20260620 23:26:59.307523 22179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217: No bootstrap required, opened a new log
I20260620 23:26:59.311853 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:59.289031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58045 (local address 127.0.97.62:41247)
0620 23:26:59.289962 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:59.289977 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:59.290495 (+ 518us) server_negotiation.cc:407] Connection header received
0620 23:26:59.292436 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:59.292461 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:59.292958 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:59.293389 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:59.294806 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:59.296806 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:59.298157 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:59.299152 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:59.299332 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:59.303405 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:59.303415 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:59.303431 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:59.303470 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:59.307349 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:59.308077 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:59.308084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:59.308091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:59.308291 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:59.308699 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:59.308706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:59.308714 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:59.309355 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:59.309690 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:59.310164 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 23:26:59.310507 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":487,"threads_started":1}
I20260620 23:26:59.315557 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:26:59.317145 22179 raft_consensus.cc:348] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd91f546d78469a86493a8e3d0ac217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41247 } }
I20260620 23:26:59.317471 22179 raft_consensus.cc:374] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:26:59.317565 22179 raft_consensus.cc:729] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd91f546d78469a86493a8e3d0ac217, State: Initialized, Role: FOLLOWER
I20260620 23:26:59.318169 22179 consensus_queue.cc:260] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd91f546d78469a86493a8e3d0ac217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41247 } }
I20260620 23:26:59.318392 22179 raft_consensus.cc:388] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:26:59.318559 22179 raft_consensus.cc:482] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:26:59.318758 22179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:26:59.321298 22179 raft_consensus.cc:504] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd91f546d78469a86493a8e3d0ac217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41247 } }
I20260620 23:26:59.321761 22179 leader_election.cc:302] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afd91f546d78469a86493a8e3d0ac217; no voters:
I20260620 23:26:59.322366 22179 leader_election.cc:288] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:26:59.322497 22189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:26:59.324410 22179 sys_catalog.cc:564] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:26:59.330986 22189 raft_consensus.cc:686] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [term 1 LEADER]: Becoming Leader. State: Replica: afd91f546d78469a86493a8e3d0ac217, State: Running, Role: LEADER
I20260620 23:26:59.332530 22189 consensus_queue.cc:237] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd91f546d78469a86493a8e3d0ac217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41247 } }
I20260620 23:26:59.338941 22190 sys_catalog.cc:455] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afd91f546d78469a86493a8e3d0ac217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd91f546d78469a86493a8e3d0ac217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41247 } } }
I20260620 23:26:59.339339 22190 sys_catalog.cc:458] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:59.339797 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afd91f546d78469a86493a8e3d0ac217. Latest consensus state: current_term: 1 leader_uuid: "afd91f546d78469a86493a8e3d0ac217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd91f546d78469a86493a8e3d0ac217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41247 } } }
I20260620 23:26:59.340057 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217 [sys.catalog]: This master's current role is: LEADER
I20260620 23:26:59.340211 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:26:59.319875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38869 (local address 127.0.97.62:41247)
0620 23:26:59.320046 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:26:59.320052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:26:59.320439 (+ 387us) server_negotiation.cc:407] Connection header received
0620 23:26:59.320534 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:26:59.320539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:26:59.320611 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:26:59.320703 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:26:59.322184 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:59.323478 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:59.325442 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:26:59.326338 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:26:59.327349 (+ 1011us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:26:59.330464 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:26:59.330473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:26:59.330481 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:26:59.330529 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:26:59.334771 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:59.336060 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:59.336069 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:59.336075 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:59.336259 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:26:59.336952 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:26:59.336960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:26:59.336966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:26:59.337422 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:26:59.337770 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 23:26:59.339369 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0620 23:26:59.339628 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:26:59.342792 22202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:26:59.347889 22202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:26:59.355819 22202 catalog_manager.cc:1347] Generated new cluster ID: f346e86bc0ad4afa8744dd9a5a798831
I20260620 23:26:59.355955 22202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:26:59.366945 22202 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:26:59.368549 22202 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:26:59.375283 22202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afd91f546d78469a86493a8e3d0ac217: Generated new TSK 0
I20260620 23:26:59.376024 22202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:26:59.381845 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22112
I20260620 23:26:59.383411 22174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:26:59.471460 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 22112
W20260620 23:26:59.487861 22097 connection.cc:535] client connection to 127.0.97.62:41247 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:41247): Connection reset by peer (error 104)
2026-06-20T23:26:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1967 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:26:59 dist-test-slave-5kzz krb5kdc[22212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:26:59 dist-test-slave-5kzz krb5kdc[22212](info): set up 2 sockets
Jun 20 23:26:59 dist-test-slave-5kzz krb5kdc[22212](info): commencing operation
krb5kdc: starting...
W20260620 23:27:00.577284 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:27:00 dist-test-slave-5kzz krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:27:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:27:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:27:00.826514 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40971
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:27:01.150854 22225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:27:01.151180 22225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:27:01.151357 22225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:27:01.158629 22225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:27:01.158855 22225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:27:01.158970 22225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:27:01.159080 22225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:27:01.159166 22225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:27:01.169171 22225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40971
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:27:01.170475 22225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:27:01.171830 22225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:27:01.183459 22233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:01.183568 22230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:01.183578 22231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:27:01.184288 22225 server_base.cc:1034] running on GCE node
I20260620 23:27:01.185034 22225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:27:01.186386 22225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:27:01.187577 22225 hybrid_clock.cc:648] HybridClock initialized: now 1781998021187517 us; error 61 us; skew 500 ppm
I20260620 23:27:01.187872 22225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:27:01 dist-test-slave-5kzz krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:27:01.191039 22225 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:27:01.192595 22225 webserver.cc:460] Webserver started at http://127.0.97.62:45435/ using document root <none> and password file <none>
I20260620 23:27:01.193262 22225 fs_manager.cc:359] Metadata directory not provided
I20260620 23:27:01.193393 22225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:27:01.193708 22225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:27:01.196785 22225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "dd9322f233fa40cfb9795675667a1012"
format_stamp: "Formatted at 2026-06-20 23:27:01 on dist-test-slave-5kzz"
server_key: "12e4f55c11b3dc651eefe2542f629447"
server_key_iv: "ed4797a9f5bd36ba9bb0589165b8aea3"
server_key_version: "encryptionkey@0"
I20260620 23:27:01.197562 22225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "dd9322f233fa40cfb9795675667a1012"
format_stamp: "Formatted at 2026-06-20 23:27:01 on dist-test-slave-5kzz"
server_key: "12e4f55c11b3dc651eefe2542f629447"
server_key_iv: "ed4797a9f5bd36ba9bb0589165b8aea3"
server_key_version: "encryptionkey@0"
I20260620 23:27:01.202881 22225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:27:01.206264 22240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:27:01.207707 22225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:27:01.207902 22225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "dd9322f233fa40cfb9795675667a1012"
format_stamp: "Formatted at 2026-06-20 23:27:01 on dist-test-slave-5kzz"
server_key: "12e4f55c11b3dc651eefe2542f629447"
server_key_iv: "ed4797a9f5bd36ba9bb0589165b8aea3"
server_key_version: "encryptionkey@0"
I20260620 23:27:01.208139 22225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:27:01.229516 22225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:27:01.233711 22225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:27:01.234020 22225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:27:01.252975 22225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40971
I20260620 23:27:01.253011 22291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40971 every 8 connection(s)
I20260620 23:27:01.254386 22225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:27:01.258049 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:27:01.261369 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 22225
I20260620 23:27:01.261545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:27:01.262077 388 external_mini_cluster.cc:1428] Setting key 38cedf763b99f64f34c5c87e0548be6d
I20260620 23:27:01.271240 22292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012: Bootstrap starting.
I20260620 23:27:01.275065 22292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012: Neither blocks nor log segments found. Creating new log.
I20260620 23:27:01.276229 22292 log.cc:824] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012: Log is configured to *not* fsync() on all Append() calls
I20260620 23:27:01.276475 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:01.264138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38163 (local address 127.0.97.62:40971)
0620 23:27:01.265528 (+ 1390us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:01.265539 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:01.265580 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:27:01.267299 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:01.267322 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:01.267745 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:01.268078 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:01.268672 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:01.269994 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:01.271905 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:01.272583 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:01.272717 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:01.274089 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:01.274097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:01.274110 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:27:01.274637 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:01.274782 (+ 145us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:27:01.274810 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:01.274938 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:01.275293 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 23:27:01.275431 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155,"thread_start_us":910,"threads_started":1}
I20260620 23:27:01.279183 22292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012: No bootstrap required, opened a new log
I20260620 23:27:01.280215 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:27:01.287772 22292 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9322f233fa40cfb9795675667a1012" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40971 } }
I20260620 23:27:01.288304 22292 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:27:01.288412 22292 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd9322f233fa40cfb9795675667a1012, State: Initialized, Role: FOLLOWER
I20260620 23:27:01.289062 22292 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9322f233fa40cfb9795675667a1012" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40971 } }
I20260620 23:27:01.289317 22292 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:27:01.289495 22292 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:27:01.289674 22292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:27:01.291466 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:01.284590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57487 (local address 127.0.97.62:40971)
0620 23:27:01.284821 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:01.284827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:01.285178 (+ 351us) server_negotiation.cc:407] Connection header received
0620 23:27:01.285369 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:01.285374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:01.285448 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:01.285537 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:01.286097 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:01.286919 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:01.288750 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:01.289266 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:01.289397 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:01.290653 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:01.290657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:01.290661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:27:01.290721 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:01.290793 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:27:01.290809 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:01.290954 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:01.291115 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 23:27:01.291169 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:27:01.292622 22292 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9322f233fa40cfb9795675667a1012" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40971 } }
I20260620 23:27:01.293155 22292 leader_election.cc:302] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd9322f233fa40cfb9795675667a1012; no voters:
I20260620 23:27:01.293731 22292 leader_election.cc:288] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:27:01.293859 22302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:27:01.294611 22302 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [term 1 LEADER]: Becoming Leader. State: Replica: dd9322f233fa40cfb9795675667a1012, State: Running, Role: LEADER
I20260620 23:27:01.295126 22302 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9322f233fa40cfb9795675667a1012" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40971 } }
I20260620 23:27:01.295984 22292 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:27:01.297968 22303 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd9322f233fa40cfb9795675667a1012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9322f233fa40cfb9795675667a1012" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40971 } } }
I20260620 23:27:01.298362 22303 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:01.299057 22304 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd9322f233fa40cfb9795675667a1012. Latest consensus state: current_term: 1 leader_uuid: "dd9322f233fa40cfb9795675667a1012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9322f233fa40cfb9795675667a1012" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40971 } } }
I20260620 23:27:01.299403 22304 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012 [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:01.300689 22307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:27:01.307184 22307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:27:01.317648 22307 catalog_manager.cc:1347] Generated new cluster ID: 48b7e93833b94c95b1d50ca2974c79fc
I20260620 23:27:01.317798 22307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:27:01.347230 22307 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:27:01.348615 22307 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:27:01.355151 22307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd9322f233fa40cfb9795675667a1012: Generated new TSK 0
I20260620 23:27:01.356042 22307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:27:01.372148 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22225
W20260620 23:27:01.373339 22287 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38163
W20260620 23:27:01.374135 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22225: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:27:01.374279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 22225
W20260620 23:27:01.391426 22210 connection.cc:535] client connection to 127.0.97.62:40971 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:40971): Connection reset by peer (error 104)
2026-06-20T23:27:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:27:01 dist-test-slave-5kzz krb5kdc[22323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:27:01 dist-test-slave-5kzz krb5kdc[22323](info): set up 2 sockets
Jun 20 23:27:01 dist-test-slave-5kzz krb5kdc[22323](info): commencing operation
krb5kdc: starting...
W20260620 23:27:02.479269 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:27:02 dist-test-slave-5kzz krb5kdc[22323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:27:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:27:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:27:02.734165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45795
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:27:03.061149 22336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:27:03.061471 22336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:27:03.061653 22336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:27:03.069178 22336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:27:03.069314 22336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:27:03.069412 22336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:27:03.069512 22336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:27:03.069625 22336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:27:03.080113 22336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45795
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:27:03.081496 22336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:27:03.082958 22336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:27:03.096711 22342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:03.096752 22344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:03.096779 22341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:27:03.098099 22336 server_base.cc:1034] running on GCE node
I20260620 23:27:03.099022 22336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:27:03.100561 22336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:27:03.101807 22336 hybrid_clock.cc:648] HybridClock initialized: now 1781998023101778 us; error 57 us; skew 500 ppm
I20260620 23:27:03.102190 22336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:27:03 dist-test-slave-5kzz krb5kdc[22323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:27:03.105417 22336 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:27:03.107159 22336 webserver.cc:460] Webserver started at http://127.0.97.62:41491/ using document root <none> and password file <none>
I20260620 23:27:03.107954 22336 fs_manager.cc:359] Metadata directory not provided
I20260620 23:27:03.108114 22336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:27:03.108525 22336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:27:03.112090 22336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "112e8474b5454a69bc9d61d99138a9dd"
format_stamp: "Formatted at 2026-06-20 23:27:03 on dist-test-slave-5kzz"
server_key: "9129a595ddbd2ea8608c390a1f020206"
server_key_iv: "8f3c02ad7865f601967f17419071edf7"
server_key_version: "encryptionkey@0"
I20260620 23:27:03.112926 22336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "112e8474b5454a69bc9d61d99138a9dd"
format_stamp: "Formatted at 2026-06-20 23:27:03 on dist-test-slave-5kzz"
server_key: "9129a595ddbd2ea8608c390a1f020206"
server_key_iv: "8f3c02ad7865f601967f17419071edf7"
server_key_version: "encryptionkey@0"
I20260620 23:27:03.118261 22336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:27:03.121783 22351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:27:03.123138 22336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:27:03.123318 22336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "112e8474b5454a69bc9d61d99138a9dd"
format_stamp: "Formatted at 2026-06-20 23:27:03 on dist-test-slave-5kzz"
server_key: "9129a595ddbd2ea8608c390a1f020206"
server_key_iv: "8f3c02ad7865f601967f17419071edf7"
server_key_version: "encryptionkey@0"
I20260620 23:27:03.123524 22336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:27:03.129385 22336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:27:03.133383 22336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:27:03.133671 22336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:27:03.151959 22336 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45795
I20260620 23:27:03.151979 22402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45795 every 8 connection(s)
I20260620 23:27:03.153398 22336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:27:03.157064 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:27:03.157670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 22336
I20260620 23:27:03.157888 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:27:03.158354 388 external_mini_cluster.cc:1428] Setting key bb038fbff79704824aa613203528282c
I20260620 23:27:03.166630 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:03.160168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50517 (local address 127.0.97.62:45795)
0620 23:27:03.161066 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:03.161077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:03.161121 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:27:03.163205 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:03.163229 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:03.163614 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:03.163999 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:03.165061 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:03.165068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:03.165080 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:27:03.165096 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:03.165155 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:27:03.165180 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:03.165369 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:03.165472 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 23:27:03.165674 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":400,"threads_started":1}
I20260620 23:27:03.170109 22403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd: Bootstrap starting.
I20260620 23:27:03.170501 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:27:03.174054 22403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd: Neither blocks nor log segments found. Creating new log.
I20260620 23:27:03.175453 22403 log.cc:824] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:27:03.178443 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:03.175823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54743 (local address 127.0.97.62:45795)
0620 23:27:03.176054 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:03.176065 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:03.176506 (+ 441us) server_negotiation.cc:407] Connection header received
0620 23:27:03.176640 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:03.176649 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:03.176722 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:03.176887 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:03.177854 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:03.177858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:03.177861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:27:03.177877 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:03.177916 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:27:03.177933 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:03.178029 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:03.178181 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 23:27:03.178229 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:27:03.179147 22403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd: No bootstrap required, opened a new log
I20260620 23:27:03.186846 22403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112e8474b5454a69bc9d61d99138a9dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45795 } }
I20260620 23:27:03.187237 22403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:27:03.187354 22403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 112e8474b5454a69bc9d61d99138a9dd, State: Initialized, Role: FOLLOWER
I20260620 23:27:03.188051 22403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112e8474b5454a69bc9d61d99138a9dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45795 } }
I20260620 23:27:03.188309 22403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:27:03.188464 22403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:27:03.188640 22403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:27:03.190797 22403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112e8474b5454a69bc9d61d99138a9dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45795 } }
I20260620 23:27:03.191453 22403 leader_election.cc:302] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 112e8474b5454a69bc9d61d99138a9dd; no voters:
I20260620 23:27:03.192157 22403 leader_election.cc:288] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:27:03.192307 22413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:27:03.192984 22413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [term 1 LEADER]: Becoming Leader. State: Replica: 112e8474b5454a69bc9d61d99138a9dd, State: Running, Role: LEADER
I20260620 23:27:03.193467 22413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112e8474b5454a69bc9d61d99138a9dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45795 } }
I20260620 23:27:03.194238 22403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:27:03.196090 22415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 112e8474b5454a69bc9d61d99138a9dd. Latest consensus state: current_term: 1 leader_uuid: "112e8474b5454a69bc9d61d99138a9dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112e8474b5454a69bc9d61d99138a9dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45795 } } }
I20260620 23:27:03.196452 22415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:03.197225 22414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "112e8474b5454a69bc9d61d99138a9dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112e8474b5454a69bc9d61d99138a9dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45795 } } }
I20260620 23:27:03.197553 22414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:03.198239 22418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:27:03.203086 22418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:27:03.211077 22418 catalog_manager.cc:1347] Generated new cluster ID: bc4c53e116c249dd9f8d724a5ba0a078
I20260620 23:27:03.211192 22418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:27:03.228240 22418 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:27:03.228751 22418 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:27:03.236613 22418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 112e8474b5454a69bc9d61d99138a9dd: Generated new TSK 0
I20260620 23:27:03.237455 22418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:27:03.242509 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22336
W20260620 23:27:03.243721 22398 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50517
W20260620 23:27:03.244483 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22336: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:27:03.244626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 22336
2026-06-20T23:27:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1870 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:27:03 dist-test-slave-5kzz krb5kdc[22434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:27:03 dist-test-slave-5kzz krb5kdc[22434](info): set up 2 sockets
Jun 20 23:27:03 dist-test-slave-5kzz krb5kdc[22434](info): commencing operation
krb5kdc: starting...
W20260620 23:27:04.347177 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:27:04 dist-test-slave-5kzz krb5kdc[22434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:27:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:27:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:27:04.581895 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33003
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:27:04.898610 22447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:27:04.898993 22447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:27:04.899150 22447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:27:04.906394 22447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:27:04.906540 22447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:27:04.906647 22447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:27:04.906775 22447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:27:04.906860 22447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:27:04.918419 22447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:27:04.919716 22447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:27:04.920989 22447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:27:04.932437 22455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:04.932557 22453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:04.932557 22452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:27:04.933602 22447 server_base.cc:1034] running on GCE node
I20260620 23:27:04.934369 22447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:27:04.935732 22447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:27:04.936929 22447 hybrid_clock.cc:648] HybridClock initialized: now 1781998024936883 us; error 43 us; skew 500 ppm
I20260620 23:27:04.937273 22447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:27:04 dist-test-slave-5kzz krb5kdc[22434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:27:04.940311 22447 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:27:04.941772 22447 webserver.cc:460] Webserver started at http://127.0.97.62:34017/ using document root <none> and password file <none>
I20260620 23:27:04.942484 22447 fs_manager.cc:359] Metadata directory not provided
I20260620 23:27:04.942617 22447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:27:04.942939 22447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:27:04.945799 22447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "3eb6f3f8f4d94b27be43454e744ef703"
format_stamp: "Formatted at 2026-06-20 23:27:04 on dist-test-slave-5kzz"
server_key: "5c23cd00cb1a6d073375e7d45568d5a7"
server_key_iv: "b1a0696ac0a29e7a838c67a7aedc7004"
server_key_version: "encryptionkey@0"
I20260620 23:27:04.946542 22447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "3eb6f3f8f4d94b27be43454e744ef703"
format_stamp: "Formatted at 2026-06-20 23:27:04 on dist-test-slave-5kzz"
server_key: "5c23cd00cb1a6d073375e7d45568d5a7"
server_key_iv: "b1a0696ac0a29e7a838c67a7aedc7004"
server_key_version: "encryptionkey@0"
I20260620 23:27:04.951617 22447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:27:04.955145 22462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:27:04.956497 22447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:27:04.956693 22447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "3eb6f3f8f4d94b27be43454e744ef703"
format_stamp: "Formatted at 2026-06-20 23:27:04 on dist-test-slave-5kzz"
server_key: "5c23cd00cb1a6d073375e7d45568d5a7"
server_key_iv: "b1a0696ac0a29e7a838c67a7aedc7004"
server_key_version: "encryptionkey@0"
I20260620 23:27:04.956900 22447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:27:04.991091 22447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:27:04.995287 22447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:27:04.995599 22447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:27:05.014336 22447 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33003
I20260620 23:27:05.014384 22513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33003 every 8 connection(s)
I20260620 23:27:05.015722 22447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:27:05.019568 22514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:27:05.023994 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 22447
I20260620 23:27:05.024187 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:27:05.024616 388 external_mini_cluster.cc:1428] Setting key 7609e72ae130472d195fcdfe7f42ff8d
I20260620 23:27:05.031800 22514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703: Bootstrap starting.
I20260620 23:27:05.035441 22514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703: Neither blocks nor log segments found. Creating new log.
I20260620 23:27:05.036746 22514 log.cc:824] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703: Log is configured to *not* fsync() on all Append() calls
I20260620 23:27:05.039674 22514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703: No bootstrap required, opened a new log
Jun 20 23:27:05 dist-test-slave-5kzz krb5kdc[22434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:27:05.050141 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:05.026529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40329 (local address 127.0.97.62:33003)
0620 23:27:05.027422 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:05.027437 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:05.027498 (+ 61us) server_negotiation.cc:407] Connection header received
0620 23:27:05.029972 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:05.030002 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:05.030515 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:05.030956 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:05.032266 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:05.034280 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:05.035627 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:05.036664 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:05.036863 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:05.041164 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:05.041174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:05.041198 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:05.041246 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:05.045428 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:05.046118 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:05.046126 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:05.046137 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:05.046356 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:05.046842 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:05.046847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:05.046851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:05.047598 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:05.047890 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:05.048282 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 23:27:05.048747 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":414,"threads_started":1}
I20260620 23:27:05.050626 22514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb6f3f8f4d94b27be43454e744ef703" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33003 } }
I20260620 23:27:05.051054 22514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:27:05.051157 22514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb6f3f8f4d94b27be43454e744ef703, State: Initialized, Role: FOLLOWER
I20260620 23:27:05.051860 22514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb6f3f8f4d94b27be43454e744ef703" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33003 } }
I20260620 23:27:05.052167 22514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:27:05.052305 22514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:27:05.052474 22514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:27:05.054407 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:27:05.055545 22514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb6f3f8f4d94b27be43454e744ef703" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33003 } }
I20260620 23:27:05.056120 22514 leader_election.cc:302] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3eb6f3f8f4d94b27be43454e744ef703; no voters:
I20260620 23:27:05.056761 22514 leader_election.cc:288] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:27:05.056881 22519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:27:05.059165 22519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [term 1 LEADER]: Becoming Leader. State: Replica: 3eb6f3f8f4d94b27be43454e744ef703, State: Running, Role: LEADER
I20260620 23:27:05.059445 22514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:27:05.059796 22519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb6f3f8f4d94b27be43454e744ef703" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33003 } }
I20260620 23:27:05.066826 22520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3eb6f3f8f4d94b27be43454e744ef703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb6f3f8f4d94b27be43454e744ef703" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33003 } } }
I20260620 23:27:05.067348 22520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:05.067922 22521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eb6f3f8f4d94b27be43454e744ef703. Latest consensus state: current_term: 1 leader_uuid: "3eb6f3f8f4d94b27be43454e744ef703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb6f3f8f4d94b27be43454e744ef703" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33003 } } }
I20260620 23:27:05.068245 22521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703 [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:05.078889 22531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:27:05.084651 22531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:27:05.086771 22541 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:27:05.091518 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:05.065686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49799 (local address 127.0.97.62:33003)
0620 23:27:05.065914 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:05.065924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:05.072939 (+ 7015us) server_negotiation.cc:407] Connection header received
0620 23:27:05.073521 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:05.073531 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:05.073645 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:05.073740 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:05.076443 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:05.077736 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:05.079756 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:05.080506 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:05.080635 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:05.083975 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:05.083982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:05.083988 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:05.084027 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:05.088129 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:05.089155 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:05.089162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:05.089165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:05.089317 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:05.089791 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:05.089797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:05.089800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:05.090239 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:05.090492 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:05.090898 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 23:27:05.091119 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:27:05.094492 22531 catalog_manager.cc:1347] Generated new cluster ID: 5310ae648bb24a6980c6ff6c45c8de5e
I20260620 23:27:05.094600 22531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:27:05.104933 22531 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:27:05.106405 22531 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:27:05.119787 22531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3eb6f3f8f4d94b27be43454e744ef703: Generated new TSK 0
I20260620 23:27:05.120867 22531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:27:05.135144 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22447
I20260620 23:27:05.136300 22509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:27:05.226589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 22447
2026-06-20T23:27:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1981 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:27:05 dist-test-slave-5kzz krb5kdc[22546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:27:05 dist-test-slave-5kzz krb5kdc[22546](info): set up 2 sockets
Jun 20 23:27:05 dist-test-slave-5kzz krb5kdc[22546](info): commencing operation
krb5kdc: starting...
W20260620 23:27:06.339202 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:27:06 dist-test-slave-5kzz krb5kdc[22546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:27:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:27:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:27:06.585346 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33063
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:27:06.909253 22559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:27:06.909571 22559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:27:06.909817 22559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:27:06.916908 22559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:27:06.917011 22559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:27:06.917097 22559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:27:06.917182 22559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:27:06.917241 22559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:27:06.926951 22559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:27:06.928242 22559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:27:06.929495 22559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:27:06.940820 22567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:06.941000 22564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:06.940912 22565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:27:06.941972 22559 server_base.cc:1034] running on GCE node
I20260620 23:27:06.942768 22559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:27:06.944090 22559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:27:06.945281 22559 hybrid_clock.cc:648] HybridClock initialized: now 1781998026945225 us; error 48 us; skew 500 ppm
I20260620 23:27:06.945614 22559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:27:06 dist-test-slave-5kzz krb5kdc[22546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:27:06.951707 22559 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:27:06.953081 22559 webserver.cc:460] Webserver started at http://127.0.97.62:33347/ using document root <none> and password file <none>
I20260620 23:27:06.953750 22559 fs_manager.cc:359] Metadata directory not provided
I20260620 23:27:06.953877 22559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:27:06.954180 22559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:27:06.957094 22559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "bf7724ad91fd40819784fbe42bfd866d"
format_stamp: "Formatted at 2026-06-20 23:27:06 on dist-test-slave-5kzz"
server_key: "2dcc7787910dd1ca12c22927dc43699f"
server_key_iv: "4fa71d9983c56b13bbf70d14b75a1151"
server_key_version: "encryptionkey@0"
I20260620 23:27:06.957906 22559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "bf7724ad91fd40819784fbe42bfd866d"
format_stamp: "Formatted at 2026-06-20 23:27:06 on dist-test-slave-5kzz"
server_key: "2dcc7787910dd1ca12c22927dc43699f"
server_key_iv: "4fa71d9983c56b13bbf70d14b75a1151"
server_key_version: "encryptionkey@0"
I20260620 23:27:06.963083 22559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:27:06.966461 22574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:27:06.967814 22559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:27:06.967991 22559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "bf7724ad91fd40819784fbe42bfd866d"
format_stamp: "Formatted at 2026-06-20 23:27:06 on dist-test-slave-5kzz"
server_key: "2dcc7787910dd1ca12c22927dc43699f"
server_key_iv: "4fa71d9983c56b13bbf70d14b75a1151"
server_key_version: "encryptionkey@0"
I20260620 23:27:06.968176 22559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:27:07.004179 22559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:27:07.008320 22559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:27:07.008639 22559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:27:07.027486 22559 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33063
I20260620 23:27:07.027519 22625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33063 every 8 connection(s)
I20260620 23:27:07.028872 22559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:27:07.029246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 22559
I20260620 23:27:07.029417 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:27:07.029959 388 external_mini_cluster.cc:1428] Setting key 07e65dadbb27fbe038e8030df66943b5
I20260620 23:27:07.033246 22626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:27:07.042611 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:07.031821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55211 (local address 127.0.97.62:33063)
0620 23:27:07.032629 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:07.032641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:07.032680 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:27:07.034342 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:07.034365 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:07.034824 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:07.035191 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:07.035732 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:07.036976 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:07.038299 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:07.038984 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:07.039173 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:07.040438 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:07.040446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:07.040463 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:27:07.041011 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:07.041085 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:27:07.041111 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:07.041267 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:07.041465 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 23:27:07.041707 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":363,"threads_started":1}
I20260620 23:27:07.044030 22626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d: Bootstrap starting.
I20260620 23:27:07.046258 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:27:07.047304 22626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d: Neither blocks nor log segments found. Creating new log.
I20260620 23:27:07.048264 22626 log.cc:824] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:27:07.051245 22626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d: No bootstrap required, opened a new log
I20260620 23:27:07.056548 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:07.050466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41693 (local address 127.0.97.62:33063)
0620 23:27:07.050671 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:07.050679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:07.050993 (+ 314us) server_negotiation.cc:407] Connection header received
0620 23:27:07.051202 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:07.051206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:07.051275 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:07.051378 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:07.051875 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:07.052592 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:07.054007 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:07.054487 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:07.054645 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:07.055802 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:07.055807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:07.055813 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:27:07.055830 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:07.055872 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:27:07.055887 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:07.055991 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:07.056140 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:27:07.056232 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:27:07.059442 22626 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7724ad91fd40819784fbe42bfd866d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33063 } }
I20260620 23:27:07.059813 22626 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:27:07.059921 22626 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7724ad91fd40819784fbe42bfd866d, State: Initialized, Role: FOLLOWER
I20260620 23:27:07.060473 22626 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7724ad91fd40819784fbe42bfd866d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33063 } }
I20260620 23:27:07.060707 22626 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:27:07.060858 22626 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:27:07.061075 22626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:27:07.063481 22626 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7724ad91fd40819784fbe42bfd866d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33063 } }
I20260620 23:27:07.063958 22626 leader_election.cc:302] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf7724ad91fd40819784fbe42bfd866d; no voters:
I20260620 23:27:07.064524 22626 leader_election.cc:288] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:27:07.064678 22636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:27:07.065399 22636 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [term 1 LEADER]: Becoming Leader. State: Replica: bf7724ad91fd40819784fbe42bfd866d, State: Running, Role: LEADER
I20260620 23:27:07.066285 22636 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7724ad91fd40819784fbe42bfd866d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33063 } }
I20260620 23:27:07.066768 22626 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:27:07.068928 22638 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf7724ad91fd40819784fbe42bfd866d. Latest consensus state: current_term: 1 leader_uuid: "bf7724ad91fd40819784fbe42bfd866d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7724ad91fd40819784fbe42bfd866d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33063 } } }
I20260620 23:27:07.069326 22638 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:07.069833 22637 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf7724ad91fd40819784fbe42bfd866d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7724ad91fd40819784fbe42bfd866d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33063 } } }
I20260620 23:27:07.070045 22637 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:07.071655 22645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:27:07.076508 22645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:27:07.084903 22645 catalog_manager.cc:1347] Generated new cluster ID: 499af75293a247729497829850fe58d5
I20260620 23:27:07.085016 22645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:27:07.109615 22645 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:27:07.111299 22645 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:27:07.121320 22645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf7724ad91fd40819784fbe42bfd866d: Generated new TSK 0
I20260620 23:27:07.122138 22645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:27:07.132404 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22559
W20260620 23:27:07.133447 22621 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55211
W20260620 23:27:07.134142 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22559: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:27:07.134294 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 22559
2026-06-20T23:27:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1907 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:27:07 dist-test-slave-5kzz krb5kdc[22657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:27:07 dist-test-slave-5kzz krb5kdc[22657](info): set up 2 sockets
Jun 20 23:27:07 dist-test-slave-5kzz krb5kdc[22657](info): commencing operation
krb5kdc: starting...
W20260620 23:27:08.239544 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:27:08 dist-test-slave-5kzz krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:27:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:27:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:27:08.492000 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44899
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:27:08.815752 22670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:27:08.816083 22670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:27:08.816267 22670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:27:08.823405 22670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:27:08.823556 22670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:27:08.823663 22670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:27:08.823781 22670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:27:08.823871 22670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:27:08.833544 22670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:27:08.834879 22670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:27:08.836174 22670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:27:08.847946 22676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:08.848011 22678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:08.848045 22675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:27:08.848809 22670 server_base.cc:1034] running on GCE node
I20260620 23:27:08.849597 22670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:27:08.851040 22670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:27:08.852273 22670 hybrid_clock.cc:648] HybridClock initialized: now 1781998028852226 us; error 42 us; skew 500 ppm
I20260620 23:27:08.852644 22670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:27:08 dist-test-slave-5kzz krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:27:08.855834 22670 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:27:08.857201 22670 webserver.cc:460] Webserver started at http://127.0.97.62:33527/ using document root <none> and password file <none>
I20260620 23:27:08.857853 22670 fs_manager.cc:359] Metadata directory not provided
I20260620 23:27:08.857995 22670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:27:08.858294 22670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:27:08.861176 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "a95facd2067a4a53907ddf6d5d9f2230"
format_stamp: "Formatted at 2026-06-20 23:27:08 on dist-test-slave-5kzz"
server_key: "06ce0b5707cdce86aa26eb9788d6bbb2"
server_key_iv: "a40f00171a6245e43cc349daef9784e6"
server_key_version: "encryptionkey@0"
I20260620 23:27:08.861935 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "a95facd2067a4a53907ddf6d5d9f2230"
format_stamp: "Formatted at 2026-06-20 23:27:08 on dist-test-slave-5kzz"
server_key: "06ce0b5707cdce86aa26eb9788d6bbb2"
server_key_iv: "a40f00171a6245e43cc349daef9784e6"
server_key_version: "encryptionkey@0"
I20260620 23:27:08.866887 22670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 23:27:08.870153 22685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:27:08.871441 22670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:27:08.871620 22670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "a95facd2067a4a53907ddf6d5d9f2230"
format_stamp: "Formatted at 2026-06-20 23:27:08 on dist-test-slave-5kzz"
server_key: "06ce0b5707cdce86aa26eb9788d6bbb2"
server_key_iv: "a40f00171a6245e43cc349daef9784e6"
server_key_version: "encryptionkey@0"
I20260620 23:27:08.871815 22670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:27:08.877210 22670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:27:08.881345 22670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:27:08.881633 22670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:27:08.899718 22670 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44899
I20260620 23:27:08.899798 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44899 every 8 connection(s)
I20260620 23:27:08.901108 22670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:27:08.904742 22737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:27:08.906514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 22670
I20260620 23:27:08.906733 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:27:08.907222 388 external_mini_cluster.cc:1428] Setting key 2ce4217d2de7e4ac800cc1bda2fc9198
I20260620 23:27:08.914654 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:08.908931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36563 (local address 127.0.97.62:44899)
0620 23:27:08.909767 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:08.909778 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:08.909816 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:27:08.911415 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:08.911436 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:08.911771 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:08.912079 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:08.913080 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:08.913086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:08.913098 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:27:08.913111 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:08.913164 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:27:08.913182 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:08.913339 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:08.913475 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 23:27:08.913666 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":396,"threads_started":1}
I20260620 23:27:08.917997 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230: Bootstrap starting.
I20260620 23:27:08.918135 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:27:08.921765 22737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230: Neither blocks nor log segments found. Creating new log.
I20260620 23:27:08.923332 22737 log.cc:824] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230: Log is configured to *not* fsync() on all Append() calls
I20260620 23:27:08.926453 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230: No bootstrap required, opened a new log
I20260620 23:27:08.928159 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:08.925566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55047 (local address 127.0.97.62:44899)
0620 23:27:08.925774 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:08.925783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:08.926262 (+ 479us) server_negotiation.cc:407] Connection header received
0620 23:27:08.926365 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:08.926371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:08.926431 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:08.926546 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:08.927517 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:08.927522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:08.927527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:27:08.927544 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:08.927587 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:27:08.927605 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:08.927732 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:08.927805 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 23:27:08.927861 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:27:08.933923 22737 raft_consensus.cc:348] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95facd2067a4a53907ddf6d5d9f2230" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44899 } }
I20260620 23:27:08.934283 22737 raft_consensus.cc:374] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:27:08.934376 22737 raft_consensus.cc:729] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95facd2067a4a53907ddf6d5d9f2230, State: Initialized, Role: FOLLOWER
I20260620 23:27:08.935005 22737 consensus_queue.cc:260] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95facd2067a4a53907ddf6d5d9f2230" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44899 } }
I20260620 23:27:08.935266 22737 raft_consensus.cc:388] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:27:08.935425 22737 raft_consensus.cc:482] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:27:08.935609 22737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:27:08.937870 22737 raft_consensus.cc:504] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95facd2067a4a53907ddf6d5d9f2230" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44899 } }
I20260620 23:27:08.938508 22737 leader_election.cc:302] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a95facd2067a4a53907ddf6d5d9f2230; no voters:
I20260620 23:27:08.939299 22737 leader_election.cc:288] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:27:08.939404 22747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:27:08.940085 22747 raft_consensus.cc:686] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [term 1 LEADER]: Becoming Leader. State: Replica: a95facd2067a4a53907ddf6d5d9f2230, State: Running, Role: LEADER
I20260620 23:27:08.940557 22747 consensus_queue.cc:237] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95facd2067a4a53907ddf6d5d9f2230" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44899 } }
I20260620 23:27:08.941282 22737 sys_catalog.cc:564] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:27:08.943248 22749 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a95facd2067a4a53907ddf6d5d9f2230. Latest consensus state: current_term: 1 leader_uuid: "a95facd2067a4a53907ddf6d5d9f2230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95facd2067a4a53907ddf6d5d9f2230" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44899 } } }
I20260620 23:27:08.943619 22749 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:08.944372 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a95facd2067a4a53907ddf6d5d9f2230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95facd2067a4a53907ddf6d5d9f2230" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44899 } } }
I20260620 23:27:08.944803 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230 [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:08.945245 22752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:27:08.950564 22752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:27:08.959488 22752 catalog_manager.cc:1347] Generated new cluster ID: e62cd0b26da84cb493a516050ba8962f
I20260620 23:27:08.959621 22752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:27:08.973340 22752 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:27:08.973742 22752 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:27:08.981510 22752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a95facd2067a4a53907ddf6d5d9f2230: Generated new TSK 0
I20260620 23:27:08.982255 22752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:27:08.987404 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22670
W20260620 23:27:08.988891 22732 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36563
W20260620 23:27:08.989609 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22670: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:27:08.989766 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 22670
2026-06-20T23:27:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1855 ms)
[----------] 18 tests from AuthTokenIssuingTest (35440 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:27:09 dist-test-slave-5kzz krb5kdc[22768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:27:09 dist-test-slave-5kzz krb5kdc[22768](info): set up 2 sockets
Jun 20 23:27:09 dist-test-slave-5kzz krb5kdc[22768](info): commencing operation
krb5kdc: starting...
W20260620 23:27:10.091280 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:27:10 dist-test-slave-5kzz krb5kdc[22768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:27:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:27:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:27:10.337914 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37121
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37121,127.0.97.61:36071,127.0.97.60:46711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:27:10.659621 22781 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:27:10.659925 22781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:27:10.659989 22781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:27:10.660072 22781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:27:10.667172 22781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:27:10.667268 22781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:27:10.667325 22781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:27:10.667372 22781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:27:10.667433 22781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:27:10.677424 22781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37121,127.0.97.61:36071,127.0.97.60:46711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:27:10.678601 22781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:27:10.679841 22781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:27:10.691305 22789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:10.691375 22787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:10.691370 22786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:27:10.692107 22781 server_base.cc:1034] running on GCE node
I20260620 23:27:10.692826 22781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:27:10.694118 22781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:27:10.695358 22781 hybrid_clock.cc:648] HybridClock initialized: now 1781998030695300 us; error 59 us; skew 500 ppm
I20260620 23:27:10.695679 22781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:27:10 dist-test-slave-5kzz krb5kdc[22768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:27:10.698597 22781 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:27:10.700028 22781 webserver.cc:460] Webserver started at http://127.0.97.62:42719/ using document root <none> and password file <none>
I20260620 23:27:10.700680 22781 fs_manager.cc:359] Metadata directory not provided
I20260620 23:27:10.700815 22781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:27:10.701129 22781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:27:10.703971 22781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "df8eaf9510b0415a9528fbb57f406f8b"
format_stamp: "Formatted at 2026-06-20 23:27:10 on dist-test-slave-5kzz"
server_key: "df85723bd86b8a8470fdf853f26bd46f"
server_key_iv: "687495c58c978e5a8179560a7dc649cc"
server_key_version: "encryptionkey@0"
I20260620 23:27:10.704736 22781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "df8eaf9510b0415a9528fbb57f406f8b"
format_stamp: "Formatted at 2026-06-20 23:27:10 on dist-test-slave-5kzz"
server_key: "df85723bd86b8a8470fdf853f26bd46f"
server_key_iv: "687495c58c978e5a8179560a7dc649cc"
server_key_version: "encryptionkey@0"
I20260620 23:27:10.709581 22781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:27:10.712862 22796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:27:10.714064 22781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:27:10.714239 22781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "df8eaf9510b0415a9528fbb57f406f8b"
format_stamp: "Formatted at 2026-06-20 23:27:10 on dist-test-slave-5kzz"
server_key: "df85723bd86b8a8470fdf853f26bd46f"
server_key_iv: "687495c58c978e5a8179560a7dc649cc"
server_key_version: "encryptionkey@0"
I20260620 23:27:10.714442 22781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:27:10.727226 22781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:27:10.731046 22781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:27:10.731325 22781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:27:10.749250 22781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37121
I20260620 23:27:10.749269 22847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37121 every 8 connection(s)
I20260620 23:27:10.750538 22781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:27:10.754271 22848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:27:10.760995 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 22781
I20260620 23:27:10.761205 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:27:10.760450 22848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:27:10.761977 388 external_mini_cluster.cc:1428] Setting key f5af5811f241a0ae5ad7d279d841fe45
Jun 20 23:27:10 dist-test-slave-5kzz krb5kdc[22768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:27:10.786206 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:10.764138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37121 (local address 127.0.0.1:37520)
0620 23:27:10.765018 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:10.765072 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:10.767788 (+ 2716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:10.768606 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:10.768615 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:10.769437 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:10.769946 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:10.769972 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:10.771829 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:10.771835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:10.773288 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:10.773300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:10.774740 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:10.774755 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:10.774885 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:10.775563 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:10.775585 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:10.777710 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:10.782131 (+ 4421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:10.782142 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:10.782152 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:10.782541 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:10.783041 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:10.783045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:10.783047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:10.783223 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:10.784270 (+ 1047us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:10.784276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:10.784469 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 23:27:10.784732 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 23:27:10.785188 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":361,"threads_started":1}
I20260620 23:27:10.786244 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:10.764138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.0.97.62:37121)
0620 23:27:10.765925 (+ 1787us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:10.765931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:10.765970 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:27:10.768145 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:10.768151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:10.768359 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:10.768510 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:10.770210 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:10.771652 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:10.773448 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:10.774456 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:10.774669 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:10.777966 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:10.777979 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:10.777998 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:10.778041 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:10.781990 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:10.782712 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:10.782723 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:10.782729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:10.782921 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:10.783343 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:10.783351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:10.783356 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:10.784129 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:10.784333 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:10.784892 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 23:27:10.785356 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":1619,"spinlock_wait_cycles":14592,"thread_start_us":1394,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260620 23:27:10.789213 22848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:27:10.790474 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:10.789965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44614)
0620 23:27:10.790218 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:10.790346 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 23:27:10.790818 22800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 23:27:10.792407 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:10.792043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44618)
0620 23:27:10.792193 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:10.792282 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:27:10.792651 22848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36071: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 23:27:10.822106 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36071
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37121,127.0.97.61:36071,127.0.97.60:46711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:27:10.842008 22848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } attempt: 1
I20260620 23:27:10.843169 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:10.842744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44632)
0620 23:27:10.842932 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:10.843022 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 23:27:10.844131 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:10.843792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44638)
0620 23:27:10.843931 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:10.844002 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:27:10.844331 22848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36071: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
I20260620 23:27:10.912592 22848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } attempt: 2
I20260620 23:27:10.913756 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:10.913276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44650)
0620 23:27:10.913526 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:10.913608 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 23:27:10.914803 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:10.914393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44652)
0620 23:27:10.914530 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:10.914601 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 23:27:10.915004 22848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36071: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
I20260620 23:27:11.006260 22848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } attempt: 3
I20260620 23:27:11.007584 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.007105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44664)
0620 23:27:11.007319 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.007432 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:27:11.008733 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.008317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44676)
0620 23:27:11.008508 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.008583 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 23:27:11.008983 22848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36071: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
W20260620 23:27:11.140311 22855 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:27:11.140625 22855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:27:11.140739 22855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:27:11.140829 22855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:27:11.147986 22855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:27:11.148082 22855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:27:11.148169 22855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:27:11.148255 22855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:27:11.148363 22855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:27:11.152253 22848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } attempt: 4
I20260620 23:27:11.153476 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.152927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44678)
0620 23:27:11.153233 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.153343 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260620 23:27:11.154511 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.154172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44694)
0620 23:27:11.154313 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.154395 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:27:11.154767 22848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36071: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36071: connect: Connection refused (error 111)
I20260620 23:27:11.158790 22855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37121,127.0.97.61:36071,127.0.97.60:46711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:27:11.160077 22855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:27:11.161326 22855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:27:11.172937 22860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:11.172937 22861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:11.173027 22863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:27:11.174369 22855 server_base.cc:1034] running on GCE node
I20260620 23:27:11.175323 22855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:27:11.176852 22855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:27:11.178149 22855 hybrid_clock.cc:648] HybridClock initialized: now 1781998031178076 us; error 41 us; skew 500 ppm
I20260620 23:27:11.178488 22855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:27:11.181700 22855 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 23:27:11.183192 22855 webserver.cc:460] Webserver started at http://127.0.97.61:44427/ using document root <none> and password file <none>
I20260620 23:27:11.183880 22855 fs_manager.cc:359] Metadata directory not provided
I20260620 23:27:11.183993 22855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:27:11.184314 22855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:27:11.187222 22855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/data/instance:
uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08"
format_stamp: "Formatted at 2026-06-20 23:27:11 on dist-test-slave-5kzz"
server_key: "3c27836dabb6f89df6a70804290f9a5c"
server_key_iv: "7740c5e76487e88ee4657788fa439b22"
server_key_version: "encryptionkey@0"
I20260620 23:27:11.187978 22855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/wal/instance:
uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08"
format_stamp: "Formatted at 2026-06-20 23:27:11 on dist-test-slave-5kzz"
server_key: "3c27836dabb6f89df6a70804290f9a5c"
server_key_iv: "7740c5e76487e88ee4657788fa439b22"
server_key_version: "encryptionkey@0"
I20260620 23:27:11.192866 22855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:27:11.196108 22870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:27:11.197427 22855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:27:11.197566 22855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/wal
uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08"
format_stamp: "Formatted at 2026-06-20 23:27:11 on dist-test-slave-5kzz"
server_key: "3c27836dabb6f89df6a70804290f9a5c"
server_key_iv: "7740c5e76487e88ee4657788fa439b22"
server_key_version: "encryptionkey@0"
I20260620 23:27:11.197803 22855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:27:11.221458 22855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:27:11.225804 22855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:27:11.226080 22855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:27:11.243829 22855 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36071
I20260620 23:27:11.243867 22921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36071 every 8 connection(s)
I20260620 23:27:11.245213 22855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
I20260620 23:27:11.245689 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 22855
I20260620 23:27:11.245886 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-1/wal/instance
I20260620 23:27:11.246495 388 external_mini_cluster.cc:1428] Setting key 160da947819cd2b7dc8d222e0325b076
I20260620 23:27:11.249013 22922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:27:11.255537 22922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:27:11.278108 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.258140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.0.97.62:37121)
0620 23:27:11.258330 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:11.258334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:11.259653 (+ 1319us) server_negotiation.cc:407] Connection header received
0620 23:27:11.261590 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:11.261596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:11.261690 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:11.261804 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:11.263372 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.264035 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.265540 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.265969 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.266118 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.272142 (+ 6024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:11.272147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:11.272151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:11.272187 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:11.274991 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.275767 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.275776 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.275779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.275899 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.276338 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.276343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.276348 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.276628 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:11.276797 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:11.277534 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 23:27:11.277781 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:27:11.279135 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.258744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37121 (local address 127.0.0.1:37526)
0620 23:27:11.259483 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.259520 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:11.261305 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:11.262039 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:11.262047 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:11.262561 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:11.263189 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.263206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.264202 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.264212 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.265393 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.265405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.266199 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.266210 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.266447 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.267172 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:11.267200 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:11.271888 (+ 4688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:11.275145 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.275155 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.275180 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.275626 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.276010 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.276015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.276019 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.276217 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.276767 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:11.276775 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:11.277136 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 23:27:11.277421 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 23:27:11.277950 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":308,"threads_started":1}
I20260620 23:27:11.280905 22922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:27:11.297326 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.281856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44698)
0620 23:27:11.282054 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.282069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:11.282196 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:11.283162 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:11.283166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:11.283585 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:11.283736 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.283744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.285093 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.285099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.286061 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.286075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.287367 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.287371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.287554 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.288180 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:11.288198 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:11.290460 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:11.294819 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.294823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.294825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.295305 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.295703 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.295708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.295712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.295878 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.296511 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:11.296514 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:11.296643 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 23:27:11.296758 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 23:27:11.296950 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":35}
I20260620 23:27:11.297493 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.281952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.0.97.61:36071)
0620 23:27:11.282620 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:11.282625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:11.282650 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:27:11.282803 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:11.282811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:11.282996 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:11.283181 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:11.283913 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.284949 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.286224 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.287203 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.287349 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.290722 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:11.290741 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:11.290764 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:11.290816 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:11.294635 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.295427 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.295434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.295436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.295554 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.295974 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.295977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.295979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.296378 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:11.296604 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:11.296895 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 23:27:11.297121 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":467,"thread_start_us":322,"threads_started":1}
I20260620 23:27:11.299968 22922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 23:27:11.301093 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.300578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35214)
0620 23:27:11.300754 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.300897 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 23:27:11.301326 22874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
I20260620 23:27:11.302201 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46711
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37121,127.0.97.61:36071,127.0.97.60:46711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:27:11.302983 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.302615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35226)
0620 23:27:11.302807 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.302889 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:27:11.303170 22922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
I20260620 23:27:11.352494 22922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } attempt: 1
I20260620 23:27:11.353677 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.353200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35232)
0620 23:27:11.353400 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.353519 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 23:27:11.354661 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.354314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35248)
0620 23:27:11.354482 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.354551 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:27:11.354931 22922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
I20260620 23:27:11.423163 22922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } attempt: 2
I20260620 23:27:11.424300 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.423850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35258)
0620 23:27:11.424041 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.424147 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 23:27:11.425246 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.424914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35272)
0620 23:27:11.425050 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.425139 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:27:11.425469 22922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
I20260620 23:27:11.454046 22848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } attempt: 5
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:27:11.469108 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.454910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.0.97.61:36071)
0620 23:27:11.455102 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:11.455107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:11.455138 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:27:11.455352 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:11.455356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:11.455432 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:11.455516 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:11.456734 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.457455 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.459220 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.459634 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.459742 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.463861 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:11.463867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:11.463870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:11.463901 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:11.466527 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.467219 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.467230 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.467232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.467348 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.467735 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.467740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.467743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.468025 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:11.468185 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:11.468570 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 23:27:11.468794 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:27:11.469112 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.454830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44708)
0620 23:27:11.455067 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.455089 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:11.455222 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:11.455590 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:11.455597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:11.456239 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:11.456524 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.456540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.457628 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.457633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.459024 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.459049 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.459807 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.459817 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.460089 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.460793 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:11.460823 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:11.463647 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:11.466694 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.466703 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.466709 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.467082 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.467463 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.467468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.467471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.467615 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.468145 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:11.468150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:11.468315 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 23:27:11.468450 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 23:27:11.468735 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:27:11.470093 22848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:27:11.471100 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.470732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35288)
0620 23:27:11.470889 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.470967 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 23:27:11.471998 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.471692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35292)
0620 23:27:11.471825 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.471894 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:27:11.472213 22848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
I20260620 23:27:11.516744 22922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } attempt: 3
I20260620 23:27:11.518242 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.517662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35294)
0620 23:27:11.517893 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.517983 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 23:27:11.519315 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.518903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35306)
0620 23:27:11.519115 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.519190 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260620 23:27:11.519587 22922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
I20260620 23:27:11.523429 22848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } attempt: 1
I20260620 23:27:11.524385 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.523997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35308)
0620 23:27:11.524158 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.524238 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:27:11.525379 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.525060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35310)
0620 23:27:11.525209 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.525282 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:27:11.525595 22848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
I20260620 23:27:11.593847 22848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } attempt: 2
I20260620 23:27:11.595104 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.594595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35314)
0620 23:27:11.594817 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.594937 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 23:27:11.596104 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.595773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35324)
0620 23:27:11.595930 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.596005 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 23:27:11.596357 22848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
W20260620 23:27:11.635697 22929 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:27:11.636034 22929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:27:11.636132 22929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:27:11.636293 22929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:27:11.643399 22929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:27:11.643481 22929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:27:11.643589 22929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:27:11.643653 22929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:27:11.643757 22929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:27:11.653436 22929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37121,127.0.97.61:36071,127.0.97.60:46711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:27:11.654752 22929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:27:11.656030 22929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:27:11.662803 22922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } attempt: 4
I20260620 23:27:11.663961 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.663469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35326)
0620 23:27:11.663726 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.663810 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 23:27:11.665046 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.664697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35328)
0620 23:27:11.664842 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.664925 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:27:11.665360 22922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
W20260620 23:27:11.668041 22934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:11.668115 22937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:11.668078 22935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:27:11.669863 22929 server_base.cc:1034] running on GCE node
I20260620 23:27:11.671003 22929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:27:11.672392 22929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:27:11.673669 22929 hybrid_clock.cc:648] HybridClock initialized: now 1781998031673568 us; error 50 us; skew 500 ppm
I20260620 23:27:11.674005 22929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:27:11.677165 22929 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 23:27:11.678747 22929 webserver.cc:460] Webserver started at http://127.0.97.60:39121/ using document root <none> and password file <none>
I20260620 23:27:11.679404 22929 fs_manager.cc:359] Metadata directory not provided
I20260620 23:27:11.679527 22929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:27:11.679831 22929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:27:11.682657 22929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/data/instance:
uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f"
format_stamp: "Formatted at 2026-06-20 23:27:11 on dist-test-slave-5kzz"
server_key: "329bbe71348f41a0f1db31afb061ca40"
server_key_iv: "4be099b1ab9278e543b1e30a30e8ccf8"
server_key_version: "encryptionkey@0"
I20260620 23:27:11.683624 22929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/wal/instance:
uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f"
format_stamp: "Formatted at 2026-06-20 23:27:11 on dist-test-slave-5kzz"
server_key: "329bbe71348f41a0f1db31afb061ca40"
server_key_iv: "4be099b1ab9278e543b1e30a30e8ccf8"
server_key_version: "encryptionkey@0"
I20260620 23:27:11.688660 22929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:27:11.692067 22944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:27:11.693452 22929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:27:11.693640 22929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/wal
uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f"
format_stamp: "Formatted at 2026-06-20 23:27:11 on dist-test-slave-5kzz"
server_key: "329bbe71348f41a0f1db31afb061ca40"
server_key_iv: "4be099b1ab9278e543b1e30a30e8ccf8"
server_key_version: "encryptionkey@0"
I20260620 23:27:11.693868 22929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:27:11.702589 22848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } attempt: 3
I20260620 23:27:11.703655 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.703191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35342)
0620 23:27:11.703416 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.703496 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:27:11.704819 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.704387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35350)
0620 23:27:11.704570 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.704690 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 23:27:11.705096 22848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46711: connect: Connection refused (error 111)
I20260620 23:27:11.709888 22929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:27:11.714046 22929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:27:11.714332 22929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:27:11.733704 22929 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46711
I20260620 23:27:11.733736 22995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46711 every 8 connection(s)
I20260620 23:27:11.735090 22929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
I20260620 23:27:11.737181 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 22929
I20260620 23:27:11.737396 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997680534866-388-0/minicluster-data/master-2/wal/instance
I20260620 23:27:11.737835 388 external_mini_cluster.cc:1428] Setting key 18b1945b1ea56b8adbf11b859a4be06a
I20260620 23:27:11.738989 22996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:27:11.747861 22996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:27:11.761629 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.739570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.0.97.60:46711)
0620 23:27:11.740305 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:11.740315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:11.740353 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:27:11.741853 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:11.741872 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:11.742225 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:11.742523 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:11.744261 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.745517 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.747263 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.747890 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.748021 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.753101 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:11.753111 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:11.753125 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:11.753174 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:11.757045 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.757905 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.757914 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.757925 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.758153 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.758591 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.758597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.758602 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.759285 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:11.759500 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:11.760011 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 23:27:11.760415 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":345,"threads_started":1}
I20260620 23:27:11.766798 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.751009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.0.97.62:37121)
0620 23:27:11.751224 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:11.751232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:11.751928 (+ 696us) server_negotiation.cc:407] Connection header received
0620 23:27:11.752228 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:11.752234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:11.752320 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:11.752424 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:11.753724 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.754561 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.756202 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.756770 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.756845 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.760797 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:11.760804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:11.760809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:11.760852 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:11.763862 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.764644 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.764649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.764652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.764872 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.765254 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.765258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.765261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.765641 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:11.765774 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:11.766318 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 23:27:11.766469 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:27:11.766947 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.750830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37121 (local address 127.0.0.1:37538)
0620 23:27:11.751800 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.751844 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:11.752058 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:11.752506 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:11.752513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:11.753264 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:11.753567 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.753576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.754768 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.754771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.755997 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.756009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.757133 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.757141 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.757345 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.758023 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:11.758047 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:11.760615 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:11.763986 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.763996 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.764001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.764504 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.765025 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.765028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.765031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.765142 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.765787 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:11.765796 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:11.766032 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 23:27:11.766229 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 23:27:11.766513 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":582,"threads_started":1}
I20260620 23:27:11.768392 22996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:27:11.779897 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.764560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.0.97.62:37121)
0620 23:27:11.765099 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:11.765103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:11.765123 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:27:11.765159 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:11.765163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:11.765247 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:11.765350 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:11.766779 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.767901 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.769243 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.769618 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.769753 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.773656 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:11.773667 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:11.773670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:11.773698 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:11.776988 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.777774 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.777781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.777787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.777931 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.778361 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.778368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.778371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.778766 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:11.779038 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:11.779265 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 23:27:11.779493 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":427,"thread_start_us":264,"threads_started":1}
I20260620 23:27:11.782279 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.768998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36071 (local address 127.0.0.1:44714)
0620 23:27:11.769152 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.769170 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:11.769258 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:11.769623 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:11.769627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:11.769965 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:11.770092 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.770099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.771114 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.771122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.771988 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.771998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.772712 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.772718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.772887 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.773397 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:11.773414 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:11.775623 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:11.779423 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.779429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.779434 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.779887 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.780621 (+ 734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.780626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.780630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.780814 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.781426 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:11.781430 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:11.781597 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 23:27:11.781767 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 23:27:11.782044 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:27:11.782950 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.769121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44714 (local address 127.0.97.61:36071)
0620 23:27:11.769325 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:11.769333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:11.769352 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:27:11.769419 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:11.769422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:11.769493 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:11.769595 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:11.770229 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.770978 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.772136 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.772560 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.772699 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.775815 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:11.775822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:11.775827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:11.775869 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:11.779276 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.780365 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.780371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.780373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.780499 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.780933 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.780942 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.780946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.781283 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:11.781431 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:11.782140 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 23:27:11.782373 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:27:11.784044 22996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:27:11.797025 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.784788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35366)
0620 23:27:11.784964 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.784981 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:11.785089 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:11.785379 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:11.785383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:11.785766 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:11.785888 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.785896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.786827 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.786834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.787659 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.787670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.788350 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.788356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.788509 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.789044 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:11.789061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:11.791869 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:11.794568 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.794575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.794579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.794986 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.795338 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.795344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.795348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.795475 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.795956 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:11.795960 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:11.796096 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 23:27:11.796249 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 23:27:11.796564 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":1}
I20260620 23:27:11.797108 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.784848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.0.97.60:46711)
0620 23:27:11.784997 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:11.785001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:11.785019 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:27:11.785194 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:11.785197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:11.785250 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:11.785324 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:11.786019 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.786728 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.787792 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.788199 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.788332 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.792030 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:11.792035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:11.792038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:11.792064 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:11.794434 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.795096 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.795104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.795107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.795228 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.795573 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.795580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.795583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.795842 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:11.795990 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:11.796327 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 23:27:11.796722 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":163,"server-negotiator.queue_time_us":50}
I20260620 23:27:11.797142 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.781567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44726 (local address 127.0.97.61:36071)
0620 23:27:11.782201 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:11.782206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:11.782227 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:27:11.782270 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:11.782276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:11.782348 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:11.782454 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:11.784176 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.785379 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.786552 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.787046 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.787205 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.791542 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:11.791548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:11.791551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:11.791581 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:11.794410 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.795130 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.795136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.795140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.795266 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.795658 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.795667 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.795669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.795921 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:11.796175 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:11.796451 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 23:27:11.796684 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":309,"threads_started":1}
I20260620 23:27:11.802472 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:27:11.803069 22996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f: Bootstrap starting.
I20260620 23:27:11.806272 22996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f: Neither blocks nor log segments found. Creating new log.
I20260620 23:27:11.807358 22996 log.cc:824] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:27:11.805173 22823 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "df8eaf9510b0415a9528fbb57f406f8b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37550
I20260620 23:27:11.810038 22996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f: No bootstrap required, opened a new log
I20260620 23:27:11.816928 22996 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } }
I20260620 23:27:11.817258 22996 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb97ad8b4563456a8b4d2dbd1cc49c5f, State: Initialized, Role: FOLLOWER
I20260620 23:27:11.817698 22996 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } }
I20260620 23:27:11.818691 23004 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } } }
I20260620 23:27:11.819109 23004 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:11.819717 22996 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:27:11.828639 23015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:27:11.828770 23015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:27:11.826751 22929 master_runner.cc:186] Error getting master registration for 127.0.97.62:37121: OK, instance_id { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" instance_seqno: 1781998030744085 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:27:11.834538 22929 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:27:11.882340 22848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } attempt: 4
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:27:11.899387 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.883150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35372)
0620 23:27:11.883351 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.883373 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:11.883509 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:11.883902 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:11.883909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:11.884454 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:11.884685 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.884700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.886139 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.886145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.887663 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.887679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.888643 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.888650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.888840 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.889537 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:11.889565 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:11.892381 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:11.896498 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.896503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.896507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.897006 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.897415 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.897421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.897424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.897607 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.898297 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:11.898302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:11.898511 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 23:27:11.898703 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 23:27:11.898986 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 23:27:11.900265 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.883213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.0.97.60:46711)
0620 23:27:11.883490 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:11.883496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:11.883520 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:27:11.883615 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:11.883621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:11.883739 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:11.883856 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:11.884855 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.885999 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.887815 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.888484 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.888609 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.892582 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:11.892589 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:11.892594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:11.892632 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:11.896311 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.897148 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.897155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.897159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.897302 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.897720 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.897724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.897728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.898169 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:11.898328 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:11.899698 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0620 23:27:11.899912 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 23:27:11.906165 22848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b: Bootstrap starting.
I20260620 23:27:11.910190 22848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b: Neither blocks nor log segments found. Creating new log.
I20260620 23:27:11.911513 22848 log.cc:824] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:27:11.914793 22848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b: No bootstrap required, opened a new log
I20260620 23:27:11.922505 22848 raft_consensus.cc:348] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } }
I20260620 23:27:11.922941 22848 raft_consensus.cc:729] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8eaf9510b0415a9528fbb57f406f8b, State: Initialized, Role: FOLLOWER
I20260620 23:27:11.923446 22848 consensus_queue.cc:260] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } }
I20260620 23:27:11.924444 23018 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } } }
I20260620 23:27:11.924908 23018 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:11.925643 22848 sys_catalog.cc:564] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:27:11.925705 22823 raft_consensus.cc:482] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:27:11.925925 22823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:27:11.928424 22823 raft_consensus.cc:504] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } }
I20260620 23:27:11.929662 22823 leader_election.cc:288] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [CANDIDATE]: Term 1 election: Requested vote from peers b4521fcf4a30466da5cc1b0d7ac5ce08 (127.0.97.61:36071), cb97ad8b4563456a8b4d2dbd1cc49c5f (127.0.97.60:46711)
I20260620 23:27:11.931301 22971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df8eaf9510b0415a9528fbb57f406f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f"
I20260620 23:27:11.931798 22971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:27:11.936450 22971 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8eaf9510b0415a9528fbb57f406f8b in term 1.
I20260620 23:27:11.937114 22800 leader_election.cc:302] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb97ad8b4563456a8b4d2dbd1cc49c5f, df8eaf9510b0415a9528fbb57f406f8b; no voters:
I20260620 23:27:11.937590 23018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:27:11.938270 23018 raft_consensus.cc:686] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [term 1 LEADER]: Becoming Leader. State: Replica: df8eaf9510b0415a9528fbb57f406f8b, State: Running, Role: LEADER
I20260620 23:27:11.938866 23018 consensus_queue.cc:237] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } }
I20260620 23:27:11.942571 23023 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8eaf9510b0415a9528fbb57f406f8b. Latest consensus state: current_term: 1 leader_uuid: "df8eaf9510b0415a9528fbb57f406f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } } }
I20260620 23:27:11.943111 23023 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:11.958659 23028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:27:11.964550 22922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } attempt: 5
W20260620 23:27:11.966560 22950 tablet.cc:2369] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 23:27:11 dist-test-slave-5kzz krb5kdc[22768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:27:11.999951 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.965379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46711 (local address 127.0.0.1:35382)
0620 23:27:11.965888 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:11.965913 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:11.966062 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:11.967535 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:11.967543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:11.968243 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:11.968516 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.968532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.970995 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.971004 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.973061 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:11.973089 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.974812 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.974822 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:11.975288 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.979947 (+ 4659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:11.979982 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:11.984338 (+ 4356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:11.992125 (+ 7787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.992134 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.992141 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.993273 (+ 1132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.994253 (+ 980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:11.994261 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:11.994267 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:11.994597 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:11.996017 (+ 1420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:11.996025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:11.996360 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 23:27:11.996635 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 23:27:11.997841 (+ 1206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337}
I20260620 23:27:12.001832 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:11.966520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.0.97.60:46711)
0620 23:27:11.966779 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:11.966788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:11.966813 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:27:11.966890 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:11.966898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:11.967003 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:11.967138 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:11.969173 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.970757 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.973327 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:11.974316 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:11.974471 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:11.984718 (+ 10247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:11.984728 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:11.984735 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:11.984790 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:11.991790 (+ 7000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.993605 (+ 1815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.993614 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.993653 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.994016 (+ 363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:11.994881 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:11.994918 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:11.994924 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:11.995796 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:11.997502 (+ 1706us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:11.998374 (+ 872us) server_negotiation.cc:299] Negotiation successful
0620 23:27:11.999474 (+ 1100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:27:12.010010 22922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08: Bootstrap starting.
W20260620 23:27:12.018487 22802 tablet.cc:2369] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:27:12.027762 22922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08: Neither blocks nor log segments found. Creating new log.
I20260620 23:27:12.035267 23028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:27:12.036302 22922 log.cc:824] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08: Log is configured to *not* fsync() on all Append() calls
I20260620 23:27:12.039606 23036 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:27:12.050689 22922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08: No bootstrap required, opened a new log
I20260620 23:27:12.061568 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.039854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.0.97.62:37121)
0620 23:27:12.040096 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.040104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.041118 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 23:27:12.042001 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.042009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.042091 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.042165 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.046558 (+ 4393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.047791 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.049949 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.050768 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.050878 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.054179 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:12.054186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:12.054190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:12.054231 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:12.058155 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:12.059064 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:12.059071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:12.059076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:12.059250 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:12.059773 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:12.059779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:12.059781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:12.060130 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:12.060366 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.060914 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.061156 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:27:12.044840 22781 master_runner.cc:186] Error getting master registration for 127.0.97.61:36071: OK, instance_id { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" instance_seqno: 1781998031238929 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:27:12.064385 22781 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:27:12.069475 22971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [term 1 FOLLOWER]: Refusing update from remote peer df8eaf9510b0415a9528fbb57f406f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:27:12.070812 23023 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:27:12.083868 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.039536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.0.97.61:36071)
0620 23:27:12.039745 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.039752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.049282 (+ 9530us) server_negotiation.cc:407] Connection header received
0620 23:27:12.049536 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.049543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.049623 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.049697 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.059953 (+ 10256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.060950 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.065234 (+ 4284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.070207 (+ 4973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.070340 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.074096 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:12.074104 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:12.074119 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:12.074161 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:12.079428 (+ 5267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:12.080597 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:12.080605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:12.080609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:12.080790 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:12.081320 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:12.081329 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:12.081334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:12.081782 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:12.082118 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.083275 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.083525 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:27:12.085058 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.041702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.0.97.60:46711)
0620 23:27:12.041939 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.041944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.060207 (+ 18263us) server_negotiation.cc:407] Connection header received
0620 23:27:12.060638 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.060644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.060728 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.060795 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.066380 (+ 5585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.067382 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.070093 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.071800 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.071894 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.076775 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:12.076783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:12.076788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:12.076828 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:12.080520 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:12.081926 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:12.081932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:12.081935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:12.082084 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:12.082890 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:12.082895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:12.082899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:12.083240 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:12.083487 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.084427 (+ 940us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.084672 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:27:12.089671 22922 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } }
I20260620 23:27:12.090134 22922 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4521fcf4a30466da5cc1b0d7ac5ce08, State: Initialized, Role: FOLLOWER
I20260620 23:27:12.090689 22922 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } }
I20260620 23:27:12.092046 23045 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } } }
I20260620 23:27:12.092737 23045 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:12.094269 22922 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:27:12.094818 22897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df8eaf9510b0415a9528fbb57f406f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08"
I20260620 23:27:12.096350 23004 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8eaf9510b0415a9528fbb57f406f8b. Latest consensus state: current_term: 1 leader_uuid: "df8eaf9510b0415a9528fbb57f406f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } } }
I20260620 23:27:12.096647 23004 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:12.094614 22896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:27:12.099146 23042 mvcc.cc:205] Tried to move back new op lower bound from 7299063939345924096 to 7299063938826457088. Current Snapshot: MvccSnapshot[applied={T|T < 7299063939345924096}]
I20260620 23:27:12.106323 22896 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [term 1 FOLLOWER]: Refusing update from remote peer df8eaf9510b0415a9528fbb57f406f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:27:12.110786 22897 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8eaf9510b0415a9528fbb57f406f8b in term 1.
I20260620 23:27:12.113422 23023 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:27:12.114498 23018 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df8eaf9510b0415a9528fbb57f406f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } } }
I20260620 23:27:12.114820 23018 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:12.118265 23028 catalog_manager.cc:1347] Generated new cluster ID: 8a88929545e74a1ab0df293a257ac756
I20260620 23:27:12.118376 23028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:27:12.119947 23004 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df8eaf9510b0415a9528fbb57f406f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } } }
I20260620 23:27:12.120208 23004 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:12.133749 23051 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df8eaf9510b0415a9528fbb57f406f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } } }
I20260620 23:27:12.134220 23051 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:12.135525 23045 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8eaf9510b0415a9528fbb57f406f8b. Latest consensus state: current_term: 1 leader_uuid: "df8eaf9510b0415a9528fbb57f406f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } } }
I20260620 23:27:12.135843 23045 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:12.157780 23023 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df8eaf9510b0415a9528fbb57f406f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8eaf9510b0415a9528fbb57f406f8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37121 } } peers { permanent_uuid: "b4521fcf4a30466da5cc1b0d7ac5ce08" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36071 } } peers { permanent_uuid: "cb97ad8b4563456a8b4d2dbd1cc49c5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46711 } } }
I20260620 23:27:12.158061 23023 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:12.168987 23028 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:27:12.171437 23028 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:27:12.195526 23028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df8eaf9510b0415a9528fbb57f406f8b: Generated new TSK 0
I20260620 23:27:12.196867 23028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:27:12.242143 23063 catalog_manager.cc:1259] Loaded cluster ID: 8a88929545e74a1ab0df293a257ac756
I20260620 23:27:12.242270 23063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08: loading cluster ID for follower catalog manager: success
I20260620 23:27:12.246196 23063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08: acquiring CA information for follower catalog manager: success
I20260620 23:27:12.248662 23063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b4521fcf4a30466da5cc1b0d7ac5ce08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 23:27:12.308414 388 mini_kdc.cc:348] Time spent kdestroy: real 0.100s user 0.003s sys 0.021s
W20260620 23:27:12.360317 23002 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:12.351293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.0.97.61:36071)
0620 23:27:12.351550 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.351560 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.357158 (+ 5598us) server_negotiation.cc:407] Connection header received
0620 23:27:12.358752 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.358760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.358849 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.358926 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.360051 (+ 1125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44736: BlockingRecv error: recv got EOF from 127.0.0.1:44736 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 23:27:12.362509 23001 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:12.342726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.0.97.62:37121)
0620 23:27:12.342918 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.342925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.356713 (+ 13788us) server_negotiation.cc:407] Connection header received
0620 23:27:12.360918 (+ 4205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.360929 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.361070 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.361151 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.362253 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37578: BlockingRecv error: recv got EOF from 127.0.0.1:37578 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 23:27:12.365372 22999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:12.357742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.0.97.60:46711)
0620 23:27:12.357905 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.357910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.363316 (+ 5406us) server_negotiation.cc:407] Connection header received
0620 23:27:12.363911 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.363917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.364006 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.364079 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.365134 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35406: BlockingRecv error: recv got EOF from 127.0.0.1:35406 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:27:12.416064 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.400322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44738 (local address 127.0.97.61:36071)
0620 23:27:12.400520 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.400527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.407553 (+ 7026us) server_negotiation.cc:407] Connection header received
0620 23:27:12.408097 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.408105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.408328 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.408472 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.409213 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.410370 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.412575 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.413352 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.413469 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.414456 (+ 987us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.415232 (+ 776us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.415366 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.415603 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.415718 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 23:27:12.426896 22999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:12.424277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.97.60:46711)
0620 23:27:12.424471 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.424479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.424972 (+ 493us) server_negotiation.cc:407] Connection header received
0620 23:27:12.425337 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.425344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.425494 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.425574 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.426625 (+ 1051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35422: BlockingRecv error: recv got EOF from 127.0.0.1:35422 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:27:12.440047 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.432630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44750 (local address 127.0.97.61:36071)
0620 23:27:12.432812 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.432820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.433297 (+ 477us) server_negotiation.cc:407] Connection header received
0620 23:27:12.433586 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.433592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.433692 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.433755 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.434365 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.435535 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.437622 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.438356 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.438496 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.439158 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.439305 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.439368 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.439624 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.439723 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 23:27:12.450290 22999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:12.447908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.0.97.60:46711)
0620 23:27:12.448087 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.448095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.448643 (+ 548us) server_negotiation.cc:407] Connection header received
0620 23:27:12.448920 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.448928 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.449023 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.449097 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.450070 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35438: BlockingRecv error: recv got EOF from 127.0.0.1:35438 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:27:12.463665 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.456440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.0.97.61:36071)
0620 23:27:12.456656 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.456663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.457121 (+ 458us) server_negotiation.cc:407] Connection header received
0620 23:27:12.457406 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.457412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.457504 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.457575 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.458234 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.459427 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.461356 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.462050 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.462185 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.462725 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.462881 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.462991 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.463193 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.463295 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 23:27:12.472908 22999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:12.470419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.0.97.60:46711)
0620 23:27:12.470599 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.470615 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.471123 (+ 508us) server_negotiation.cc:407] Connection header received
0620 23:27:12.471457 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.471463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.471555 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.471645 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.472647 (+ 1002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35440: BlockingRecv error: recv got EOF from 127.0.0.1:35440 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:27:12.488044 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.481074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44754 (local address 127.0.97.61:36071)
0620 23:27:12.481308 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.481315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.481803 (+ 488us) server_negotiation.cc:407] Connection header received
0620 23:27:12.481992 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.481998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.482099 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.482205 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.482705 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.483936 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.485923 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.486580 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.486785 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.487229 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.487400 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.487500 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.487655 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.487739 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260620 23:27:12.509145 22999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:12.500480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.0.97.60:46711)
0620 23:27:12.500774 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.500780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.507323 (+ 6543us) server_negotiation.cc:407] Connection header received
0620 23:27:12.507641 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.507648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.507761 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.507840 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.508912 (+ 1072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38186: BlockingRecv error: recv got EOF from 127.0.0.1:38186 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
I20260620 23:27:12.528936 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.521685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.0.97.61:36071)
0620 23:27:12.521893 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.521899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.522545 (+ 646us) server_negotiation.cc:407] Connection header received
0620 23:27:12.522748 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.522755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.522893 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.523002 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.523452 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.524665 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.526662 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.527425 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.527574 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.528067 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.528243 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.528345 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.528510 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.528603 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 23:27:12.543789 22999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:12.541175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.0.97.60:46711)
0620 23:27:12.541366 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.541373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.542242 (+ 869us) server_negotiation.cc:407] Connection header received
0620 23:27:12.542446 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.542452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.542546 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.542664 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.543543 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38202: BlockingRecv error: recv got EOF from 127.0.0.1:38202 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:27:12.572098 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.564804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.0.97.61:36071)
0620 23:27:12.564996 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.565002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.565549 (+ 547us) server_negotiation.cc:407] Connection header received
0620 23:27:12.565801 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.565807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.565899 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.566005 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.566468 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.567703 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.569715 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.570421 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.570710 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.571221 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.571387 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.571498 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.571712 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.571800 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 23:27:12.581173 22999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:12.578944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.0.97.60:46711)
0620 23:27:12.579118 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.579124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.579585 (+ 461us) server_negotiation.cc:407] Connection header received
0620 23:27:12.579862 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.579868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.579956 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.580024 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.580950 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38208: BlockingRecv error: recv got EOF from 127.0.0.1:38208 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:27:12.627543 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.620094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.0.97.61:36071)
0620 23:27:12.620298 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.620305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.620837 (+ 532us) server_negotiation.cc:407] Connection header received
0620 23:27:12.621123 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.621148 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.621240 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.621335 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.622002 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.623222 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.625258 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.625891 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.625999 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.626576 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.626793 (+ 217us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.626872 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.627093 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.627189 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 23:27:12.636353 22999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:12.634290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.0.97.60:46711)
0620 23:27:12.634475 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.634482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.634961 (+ 479us) server_negotiation.cc:407] Connection header received
0620 23:27:12.635094 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.635100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.635204 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.635320 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.636087 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38212: BlockingRecv error: recv got EOF from 127.0.0.1:38212 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:27:12.710237 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.704049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.0.97.61:36071)
0620 23:27:12.704283 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.704290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.704543 (+ 253us) server_negotiation.cc:407] Connection header received
0620 23:27:12.704725 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.704732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.704834 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.704959 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.705325 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.706494 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.708051 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.708746 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.708891 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.709335 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.709523 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.709640 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.709819 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.709909 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 23:27:12.718134 22999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:12.716250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.0.97.60:46711)
0620 23:27:12.716438 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.716445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.716734 (+ 289us) server_negotiation.cc:407] Connection header received
0620 23:27:12.716883 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.716888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.716982 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.717104 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:12.717885 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38222: BlockingRecv error: recv got EOF from 127.0.0.1:38222 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:27:12.830797 23015 catalog_manager.cc:1259] Loaded cluster ID: 8a88929545e74a1ab0df293a257ac756
I20260620 23:27:12.830905 23015 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f: loading cluster ID for follower catalog manager: success
I20260620 23:27:12.833365 23015 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f: acquiring CA information for follower catalog manager: success
I20260620 23:27:12.835181 23015 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cb97ad8b4563456a8b4d2dbd1cc49c5f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:27:12.854985 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.849705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.0.97.61:36071)
0620 23:27:12.849971 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.849979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.850260 (+ 281us) server_negotiation.cc:407] Connection header received
0620 23:27:12.850504 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.850510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.850586 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.850718 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.851130 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.851877 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.853244 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.853698 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.853838 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.854252 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.854361 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.854461 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.854628 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.854735 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 23:27:12.866403 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.860737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.0.97.60:46711)
0620 23:27:12.860963 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.860969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.861165 (+ 196us) server_negotiation.cc:407] Connection header received
0620 23:27:12.861375 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.861381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.861571 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.861700 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.862104 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.862935 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.864232 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.864644 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.864786 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.865280 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.865795 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.865963 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.866014 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.866094 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:27:12.880262 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.873869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.0.97.60:46711)
0620 23:27:12.874026 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.874031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.874544 (+ 513us) server_negotiation.cc:407] Connection header received
0620 23:27:12.874829 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.874837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.874932 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.875034 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.875560 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.876521 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.878123 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.878807 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.878936 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.879497 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.879635 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.879735 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.879890 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.880007 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:27:12.880369 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.872459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.0.97.61:36071)
0620 23:27:12.872781 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.872788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.873733 (+ 945us) server_negotiation.cc:407] Connection header received
0620 23:27:12.873998 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.874003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.874112 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.874184 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.875015 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.875912 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.878232 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.878880 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.879009 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.879580 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.879701 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.879786 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.879953 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.880061 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260620 23:27:12.881264 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:12.872348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.0.97.62:37121)
0620 23:27:12.873192 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:12.873197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:12.873813 (+ 616us) server_negotiation.cc:407] Connection header received
0620 23:27:12.873968 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:12.873976 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:12.874236 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:12.874434 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:12.875190 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.876233 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.878470 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:12.879013 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:12.879149 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:12.879989 (+ 840us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:12.880453 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:12.880562 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:12.880714 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 23:27:12.880940 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":571,"threads_started":1}
I20260620 23:27:12.886024 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22781
I20260620 23:27:12.887105 22843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:27:12.997390 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 22781
I20260620 23:27:13.018848 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22855
I20260620 23:27:13.019989 22917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:27:13.127439 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 22855
I20260620 23:27:13.149204 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 22929
I20260620 23:27:13.150235 22991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:27:13.257447 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 22929
2026-06-20T23:27:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4273 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:27:13 dist-test-slave-5kzz krb5kdc[23178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:27:13 dist-test-slave-5kzz krb5kdc[23178](info): set up 2 sockets
Jun 20 23:27:13 dist-test-slave-5kzz krb5kdc[23178](info): commencing operation
krb5kdc: starting...
W20260620 23:27:14.399506 388 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/test-user.keytab.
Jun 20 23:27:14 dist-test-slave-5kzz krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:27:14.682948 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38773
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38773,127.0.97.61:36039,127.0.97.60:44115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:27:15.018285 23191 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:27:15.018728 23191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:27:15.018867 23191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:27:15.019024 23191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:27:15.026464 23191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:27:15.026628 23191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:27:15.026760 23191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:27:15.026857 23191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:27:15.026948 23191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:27:15.037442 23191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38773,127.0.97.61:36039,127.0.97.60:44115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:27:15.038872 23191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:27:15.040262 23191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:27:15.053212 23199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:15.053346 23197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:15.053347 23196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:27:15.054342 23191 server_base.cc:1034] running on GCE node
I20260620 23:27:15.055223 23191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:27:15.056684 23191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:27:15.057915 23191 hybrid_clock.cc:648] HybridClock initialized: now 1781998035057848 us; error 45 us; skew 500 ppm
I20260620 23:27:15.058295 23191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:27:15 dist-test-slave-5kzz krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:27:15.061882 23191 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:27:15.063553 23191 webserver.cc:460] Webserver started at http://127.0.97.62:40717/ using document root <none> and password file <none>
I20260620 23:27:15.064283 23191 fs_manager.cc:359] Metadata directory not provided
I20260620 23:27:15.064426 23191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:27:15.064766 23191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:27:15.067948 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/data/instance:
uuid: "2eed25a5b096434fba55e066b9c33ead"
format_stamp: "Formatted at 2026-06-20 23:27:15 on dist-test-slave-5kzz"
server_key: "9cb666d0d09f93e01600969f80e838d8"
server_key_iv: "fb3d861c83b106ed7c01dcf453960b69"
server_key_version: "encryptionkey@0"
I20260620 23:27:15.068783 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/wal/instance:
uuid: "2eed25a5b096434fba55e066b9c33ead"
format_stamp: "Formatted at 2026-06-20 23:27:15 on dist-test-slave-5kzz"
server_key: "9cb666d0d09f93e01600969f80e838d8"
server_key_iv: "fb3d861c83b106ed7c01dcf453960b69"
server_key_version: "encryptionkey@0"
I20260620 23:27:15.074247 23191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:27:15.078083 23206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:27:15.079691 23191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:27:15.079906 23191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/wal
uuid: "2eed25a5b096434fba55e066b9c33ead"
format_stamp: "Formatted at 2026-06-20 23:27:15 on dist-test-slave-5kzz"
server_key: "9cb666d0d09f93e01600969f80e838d8"
server_key_iv: "fb3d861c83b106ed7c01dcf453960b69"
server_key_version: "encryptionkey@0"
I20260620 23:27:15.080164 23191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:27:15.098014 23191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:27:15.102363 23191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:27:15.102663 23191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:27:15.123116 23191 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38773
I20260620 23:27:15.123124 23257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38773 every 8 connection(s)
I20260620 23:27:15.124512 23191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:27:15.128314 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:27:15.130954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 23191
I20260620 23:27:15.131379 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-0/wal/instance
I20260620 23:27:15.131919 388 external_mini_cluster.cc:1428] Setting key b69c4cfafab5b9ca3c2abcb5aac212f2
I20260620 23:27:15.135301 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:27:15 dist-test-slave-5kzz krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 23:27:15.163216 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.138582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.0.97.62:38773)
0620 23:27:15.139500 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:15.139523 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:15.139986 (+ 463us) server_negotiation.cc:407] Connection header received
0620 23:27:15.143288 (+ 3302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:15.143299 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:15.143680 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:15.143877 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:15.146095 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.147481 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.149798 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.150777 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.150972 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:15.154641 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:15.154654 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:15.154699 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:15.154743 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:15.158888 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:15.159618 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:15.159628 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:15.159634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:15.159815 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:15.160268 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:15.160276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:15.160282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:15.160820 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:15.160973 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:15.161673 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 23:27:15.162071 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":173,"server-negotiator.queue_time_us":665,"thread_start_us":398,"threads_started":1}
I20260620 23:27:15.163313 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.138582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38773 (local address 127.0.0.1:35092)
0620 23:27:15.139803 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.139844 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:15.142621 (+ 2777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:15.144492 (+ 1871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:15.144501 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:15.145335 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:15.145843 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:15.145878 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.147676 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.147684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:15.149624 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:15.149643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.151016 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.151021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:15.151151 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:15.151870 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:15.151895 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:15.154381 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:15.159049 (+ 4668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:15.159066 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:15.159077 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:15.159464 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:15.159932 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:15.159935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:15.159937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:15.160156 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:15.160966 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:15.160973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:15.161294 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 23:27:15.161519 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 23:27:15.161956 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":3,"thread_start_us":763,"threads_started":1}
I20260620 23:27:15.166195 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:27:15.167544 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.167018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60500)
0620 23:27:15.167228 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.167366 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 23:27:15.167807 23210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
I20260620 23:27:15.169476 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.169021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60516)
0620 23:27:15.169215 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.169309 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 23:27:15.169723 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36039: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 23:27:15.194871 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36039
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38773,127.0.97.61:36039,127.0.97.60:44115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:27:15.219121 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } attempt: 1
I20260620 23:27:15.220278 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.219838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60520)
0620 23:27:15.220039 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.220136 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 23:27:15.221374 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.221007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60526)
0620 23:27:15.221187 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.221253 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 23:27:15.221561 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36039: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
I20260620 23:27:15.289832 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } attempt: 2
I20260620 23:27:15.291181 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.290640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60534)
0620 23:27:15.290923 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.291011 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 23:27:15.292230 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.291861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60546)
0620 23:27:15.292018 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.292092 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:27:15.292414 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36039: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
I20260620 23:27:15.383698 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } attempt: 3
I20260620 23:27:15.384852 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.384421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60556)
0620 23:27:15.384626 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.384717 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 23:27:15.386036 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.385628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60562)
0620 23:27:15.385810 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.385888 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 23:27:15.386264 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36039: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
W20260620 23:27:15.527875 23265 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:27:15.528227 23265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:27:15.528343 23265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:27:15.528467 23265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 23:27:15.529552 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } attempt: 4
I20260620 23:27:15.530797 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.530218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60576)
0620 23:27:15.530512 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.530609 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260620 23:27:15.532161 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.531681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60586)
0620 23:27:15.531873 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.532001 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 23:27:15.532399 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36039: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36039: connect: Connection refused (error 111)
W20260620 23:27:15.537161 23265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:27:15.537259 23265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:27:15.537369 23265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:27:15.537427 23265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:27:15.537509 23265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:27:15.547915 23265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38773,127.0.97.61:36039,127.0.97.60:44115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:27:15.549276 23265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:27:15.550611 23265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:27:15.563323 23270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:15.563243 23273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:15.563302 23271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:27:15.571666 23265 server_base.cc:1034] running on GCE node
I20260620 23:27:15.573170 23265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:27:15.575632 23265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:27:15.576896 23265 hybrid_clock.cc:648] HybridClock initialized: now 1781998035576808 us; error 58 us; skew 500 ppm
I20260620 23:27:15.577490 23265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:27:15 dist-test-slave-5kzz krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:27:15.581921 23265 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 23:27:15.583537 23265 webserver.cc:460] Webserver started at http://127.0.97.61:32805/ using document root <none> and password file <none>
I20260620 23:27:15.584307 23265 fs_manager.cc:359] Metadata directory not provided
I20260620 23:27:15.584439 23265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:27:15.584820 23265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:27:15.587903 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/data/instance:
uuid: "76708ab1fa454894a3a810f3b742340a"
format_stamp: "Formatted at 2026-06-20 23:27:15 on dist-test-slave-5kzz"
server_key: "1fdd5f28f33d00ba3b7636f666116c84"
server_key_iv: "e31abd51599cfc7b4215ca16c5b45003"
server_key_version: "encryptionkey@0"
I20260620 23:27:15.588713 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/wal/instance:
uuid: "76708ab1fa454894a3a810f3b742340a"
format_stamp: "Formatted at 2026-06-20 23:27:15 on dist-test-slave-5kzz"
server_key: "1fdd5f28f33d00ba3b7636f666116c84"
server_key_iv: "e31abd51599cfc7b4215ca16c5b45003"
server_key_version: "encryptionkey@0"
I20260620 23:27:15.593932 23265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:27:15.597666 23280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:27:15.599069 23265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:27:15.599249 23265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/wal
uuid: "76708ab1fa454894a3a810f3b742340a"
format_stamp: "Formatted at 2026-06-20 23:27:15 on dist-test-slave-5kzz"
server_key: "1fdd5f28f33d00ba3b7636f666116c84"
server_key_iv: "e31abd51599cfc7b4215ca16c5b45003"
server_key_version: "encryptionkey@0"
I20260620 23:27:15.599467 23265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:27:15.616551 23265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:27:15.620723 23265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:27:15.621071 23265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:27:15.639333 23265 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36039
I20260620 23:27:15.639354 23331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36039 every 8 connection(s)
I20260620 23:27:15.640697 23265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/data/info.pb
I20260620 23:27:15.641672 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 23265
I20260620 23:27:15.641856 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-1/wal/instance
I20260620 23:27:15.642431 388 external_mini_cluster.cc:1428] Setting key 35f77502d9172a90115c1cdc4c3b46ae
I20260620 23:27:15.645442 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:27:15.655131 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:27:15 dist-test-slave-5kzz krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 23:27:15.675987 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.657884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.0.97.62:38773)
0620 23:27:15.658077 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:15.658082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:15.659510 (+ 1428us) server_negotiation.cc:407] Connection header received
0620 23:27:15.661491 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:15.661498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:15.661599 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:15.661709 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:15.663277 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.663970 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.665503 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.665922 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.666072 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:15.669793 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:15.669798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:15.669801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:15.669835 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:15.672841 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:15.673615 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:15.673619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:15.673622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:15.673739 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:15.674188 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:15.674192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:15.674194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:15.674472 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:15.674635 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:15.675427 (+ 792us) server_negotiation.cc:299] Negotiation successful
0620 23:27:15.675658 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:27:15.677033 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.658494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38773 (local address 127.0.0.1:35100)
0620 23:27:15.659342 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.659383 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:15.661211 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:15.661945 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:15.661954 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:15.662481 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:15.663102 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:15.663120 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.664145 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.664149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:15.665346 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:15.665358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.666150 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.666157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:15.666390 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:15.667110 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:15.667132 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:15.669521 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:15.672997 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:15.673007 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:15.673025 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:15.673459 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:15.673856 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:15.673861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:15.673865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:15.674052 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:15.674607 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:15.674615 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:15.675049 (+ 434us) client_negotiation.cc:769] Sending connection context
0620 23:27:15.675316 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 23:27:15.675855 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":387,"threads_started":1}
I20260620 23:27:15.678952 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:27:15 dist-test-slave-5kzz krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:27:15.696393 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.679889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60600)
0620 23:27:15.680104 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.680122 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:15.680241 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:15.681265 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:15.681272 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:15.681794 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:15.681999 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:15.682013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.683485 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.683493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:15.685057 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:15.685073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.685914 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.685926 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:15.686164 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:15.686810 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:15.686837 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:15.689299 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:15.693676 (+ 4377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:15.693681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:15.693684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:15.694157 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:15.694783 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:15.694787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:15.694789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:15.694915 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:15.695682 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:15.695685 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:15.695815 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 23:27:15.695926 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 23:27:15.696125 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 23:27:15.697036 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.680002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.0.97.61:36039)
0620 23:27:15.680840 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:15.680845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:15.680869 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:27:15.680913 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:15.680918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:15.681092 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:15.681205 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:15.682169 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.683331 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.685199 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.685765 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.685867 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:15.689516 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:15.689532 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:15.689552 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:15.689601 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:15.693511 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:15.694307 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:15.694319 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:15.694325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:15.694477 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:15.695045 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:15.695051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:15.695055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:15.695539 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:15.695774 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:15.696124 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 23:27:15.696402 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":527,"threads_started":1}
I20260620 23:27:15.699513 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:27:15.700767 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.700168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35222)
0620 23:27:15.700396 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.700551 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 23:27:15.701068 23283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
I20260620 23:27:15.702742 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.702277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35238)
0620 23:27:15.702434 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.702537 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 23:27:15.703030 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44115: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 23:27:15.706266 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaeh70X/build/asan/bin/kudu
/tmp/dist-test-taskaeh70X/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44115
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38773,127.0.97.61:36039,127.0.97.60:44115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:27:15.752352 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } attempt: 1
I20260620 23:27:15.753633 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.753167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35244)
0620 23:27:15.753376 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.753474 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:27:15.754787 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.754366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35254)
0620 23:27:15.754510 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.754624 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:27:15.754969 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44115: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
I20260620 23:27:15.823201 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } attempt: 2
I20260620 23:27:15.824530 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.823958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35268)
0620 23:27:15.824200 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.824321 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 23:27:15.825618 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.825227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35278)
0620 23:27:15.825382 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.825476 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:27:15.825843 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44115: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
I20260620 23:27:15.831712 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } attempt: 5
Jun 20 23:27:15 dist-test-slave-5kzz krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:27:15.846167 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.832565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60614)
0620 23:27:15.832784 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.832805 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:15.832973 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:15.833356 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:15.833360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:15.833773 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:15.833940 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:15.833948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.834985 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.834995 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:15.835928 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:15.835939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.836690 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.836695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:15.836897 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:15.837520 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:15.837540 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:15.839594 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:15.843556 (+ 3962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:15.843561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:15.843563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:15.843972 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:15.844413 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:15.844417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:15.844419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:15.844586 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:15.845247 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:15.845251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:15.845399 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 23:27:15.845568 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 23:27:15.845908 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 23:27:15.846251 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.832596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.0.97.61:36039)
0620 23:27:15.832785 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:15.832791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:15.832894 (+ 103us) server_negotiation.cc:407] Connection header received
0620 23:27:15.833124 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:15.833132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:15.833214 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:15.833323 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:15.834088 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.834837 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.836069 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:15.836518 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:15.836681 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:15.839821 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:15.839831 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:15.839839 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:15.839890 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:15.843362 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:15.844103 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:15.844109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:15.844113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:15.844261 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:15.844718 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:15.844726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:15.844731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:15.845103 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:15.845256 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:15.845670 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 23:27:15.845907 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:27:15.847311 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:27:15.848549 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.848183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35288)
0620 23:27:15.848323 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.848406 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 23:27:15.849527 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.849207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35290)
0620 23:27:15.849342 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.849429 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 23:27:15.849750 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44115: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
I20260620 23:27:15.901007 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } attempt: 1
I20260620 23:27:15.902235 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.901767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35302)
0620 23:27:15.901969 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.902067 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:27:15.903281 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.902915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35318)
0620 23:27:15.903087 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.903167 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:27:15.903497 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44115: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
I20260620 23:27:15.917100 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } attempt: 3
I20260620 23:27:15.918401 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.917874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35320)
0620 23:27:15.918106 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.918211 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:27:15.919729 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.919151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35324)
0620 23:27:15.919438 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.919551 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260620 23:27:15.920018 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44115: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
I20260620 23:27:15.971755 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } attempt: 2
I20260620 23:27:15.972894 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.972420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35338)
0620 23:27:15.972615 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.972726 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 23:27:15.974020 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:15.973594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35344)
0620 23:27:15.973775 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:15.973875 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 23:27:15.974493 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44115: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
W20260620 23:27:16.042984 23339 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:27:16.043316 23339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:27:16.043380 23339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:27:16.043469 23339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:27:16.050838 23339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:27:16.050932 23339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:27:16.050992 23339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:27:16.051043 23339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:27:16.051110 23339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:27:16.061007 23339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38773,127.0.97.61:36039,127.0.97.60:44115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:27:16.062302 23339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:27:16.063290 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } attempt: 4
I20260620 23:27:16.063755 23339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:27:16.064507 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.063975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35350)
0620 23:27:16.064215 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:16.064324 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 23:27:16.065651 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.065246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35364)
0620 23:27:16.065423 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:16.065506 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 23:27:16.065908 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44115: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
W20260620 23:27:16.076560 23345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:16.076650 23347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:27:16.076663 23344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:27:16.077997 23339 server_base.cc:1034] running on GCE node
I20260620 23:27:16.078862 23339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:27:16.080376 23339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:27:16.080716 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } attempt: 3
I20260620 23:27:16.081607 23339 hybrid_clock.cc:648] HybridClock initialized: now 1781998036081539 us; error 46 us; skew 500 ppm
I20260620 23:27:16.081857 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.081281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35376)
0620 23:27:16.081541 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:16.081629 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 23:27:16.082019 23339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:27:16.083009 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.082544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35378)
0620 23:27:16.082780 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:16.082867 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260620 23:27:16.083338 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44115: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44115: connect: Connection refused (error 111)
Jun 20 23:27:16 dist-test-slave-5kzz krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:27:16.085534 23339 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 23:27:16.087028 23339 webserver.cc:460] Webserver started at http://127.0.97.60:33667/ using document root <none> and password file <none>
I20260620 23:27:16.087725 23339 fs_manager.cc:359] Metadata directory not provided
I20260620 23:27:16.087857 23339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:27:16.088186 23339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:27:16.091243 23339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/data/instance:
uuid: "170171543817485dbf7e54b53d6cfec8"
format_stamp: "Formatted at 2026-06-20 23:27:16 on dist-test-slave-5kzz"
server_key: "3a9d947607a3676af9c828f01c9e2d64"
server_key_iv: "acdd9b8234dead4d0e306e8b4415408d"
server_key_version: "encryptionkey@0"
I20260620 23:27:16.092033 23339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/wal/instance:
uuid: "170171543817485dbf7e54b53d6cfec8"
format_stamp: "Formatted at 2026-06-20 23:27:16 on dist-test-slave-5kzz"
server_key: "3a9d947607a3676af9c828f01c9e2d64"
server_key_iv: "acdd9b8234dead4d0e306e8b4415408d"
server_key_version: "encryptionkey@0"
I20260620 23:27:16.097095 23339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:27:16.100871 23354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:27:16.102470 23339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:27:16.102772 23339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/wal
uuid: "170171543817485dbf7e54b53d6cfec8"
format_stamp: "Formatted at 2026-06-20 23:27:16 on dist-test-slave-5kzz"
server_key: "3a9d947607a3676af9c828f01c9e2d64"
server_key_iv: "acdd9b8234dead4d0e306e8b4415408d"
server_key_version: "encryptionkey@0"
I20260620 23:27:16.103114 23339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:27:16.120994 23339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:27:16.125531 23339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:27:16.125834 23339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:27:16.145126 23339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44115
I20260620 23:27:16.145123 23405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44115 every 8 connection(s)
I20260620 23:27:16.147142 23339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/data/info.pb
I20260620 23:27:16.151360 23406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:27:16.151655 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaeh70X/build/asan/bin/kudu as pid 23339
I20260620 23:27:16.151827 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaeh70X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997680534866-388-0/minicluster-data/master-2/wal/instance
I20260620 23:27:16.152253 388 external_mini_cluster.cc:1428] Setting key 10b7be5c2d894d40d3e202da36b4074e
I20260620 23:27:16.161412 23406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:27:16 dist-test-slave-5kzz krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 23:27:16 dist-test-slave-5kzz krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:27:16.176373 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.154103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.0.97.60:44115)
0620 23:27:16.154947 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.154962 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.155012 (+ 50us) server_negotiation.cc:407] Connection header received
0620 23:27:16.156609 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.156640 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.157171 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.157630 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.159170 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.161031 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.162443 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.163439 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.163642 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.167859 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:16.167867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:16.167877 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:16.167912 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:16.171466 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.172192 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.172202 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.172217 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.172389 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.172779 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.172784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.172789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.173535 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:16.173848 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:16.174409 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 23:27:16.174893 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":353,"threads_started":1}
I20260620 23:27:16.179575 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.164154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38773 (local address 127.0.0.1:35102)
0620 23:27:16.164815 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:16.164848 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:16.165019 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:16.165405 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:16.165412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:16.166014 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:16.166257 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:16.166265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.167398 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.167403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:16.168482 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:16.168498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.169743 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.169751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:16.169970 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.170584 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:16.170602 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:16.172752 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:16.176828 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:16.176836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:16.176841 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:16.177248 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:16.177695 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:16.177698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:16.177700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:16.177880 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:16.178426 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:16.178433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:16.178649 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 23:27:16.178890 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 23:27:16.179201 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":349,"threads_started":1}
I20260620 23:27:16.179659 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.164192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.0.97.62:38773)
0620 23:27:16.164463 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.164472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.164939 (+ 467us) server_negotiation.cc:407] Connection header received
0620 23:27:16.165162 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.165174 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.165237 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.165342 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.166437 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.167250 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.168751 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.169439 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.169622 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.172944 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:16.172951 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:16.172956 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:16.172999 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:16.176670 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.177374 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.177380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.177383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.177556 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.178016 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.178020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.178022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.178299 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:16.178418 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:16.179030 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 23:27:16.179264 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:27:16.181384 23406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:27:16 dist-test-slave-5kzz krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 23:27:16 dist-test-slave-5kzz krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:27:16.195431 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.182296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36039 (local address 127.0.0.1:60618)
0620 23:27:16.182459 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:16.182491 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:16.182615 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:16.182968 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:16.182973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:16.183354 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:16.183551 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:16.183559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.184572 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.184577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:16.185497 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:16.185508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.186208 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.186214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:16.186388 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.186971 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:16.186989 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:16.189623 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:16.192818 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:16.192824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:16.192828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:16.193225 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:16.193646 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:16.193651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:16.193654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:16.193803 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:16.194324 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:16.194330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:16.194493 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 23:27:16.194635 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 23:27:16.194980 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 23:27:16.195462 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.180070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.0.97.62:38773)
0620 23:27:16.180227 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.180232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.180249 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:27:16.180385 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.180392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.180454 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.180569 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.181955 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.183040 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.184588 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.185005 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.185144 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.189624 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:16.189629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:16.189633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:16.189668 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:16.192601 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.193383 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.193389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.193392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.193535 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.193982 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.193997 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.194003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.194307 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:16.194627 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:16.194905 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 23:27:16.195137 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:27:16.195463 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.182469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.0.97.61:36039)
0620 23:27:16.182668 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.182701 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.182720 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:27:16.182782 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.182785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.182842 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.182949 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.183692 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.184411 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.185637 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.186057 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.186192 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.189842 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:16.189848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:16.189851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:16.189887 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:16.192648 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.193358 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.193364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.193367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.193520 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.193912 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.193919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.193921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.194195 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:16.194363 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:16.194762 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 23:27:16.194983 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:27:16.196750 23406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:27:16 dist-test-slave-5kzz krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 23:27:16 dist-test-slave-5kzz krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:27:16.211938 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.197720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.0.97.60:44115)
0620 23:27:16.197892 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.197898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.197917 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:27:16.198030 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.198033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.198085 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.198156 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.199144 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.200137 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.201869 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.202439 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.202568 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.206220 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:16.206228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:16.206234 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:16.206277 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:16.208954 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.209734 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.209742 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.209745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.209873 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.210337 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.210345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.210349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.210755 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:16.210912 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:16.211311 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 23:27:16.211563 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:27:16.212136 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.197636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35402)
0620 23:27:16.197812 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:16.197831 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:16.197926 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:16.198217 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:16.198222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:16.198777 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:16.198980 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:16.199007 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.200336 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.200340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:16.201722 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:16.201736 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.202600 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.202605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:16.202821 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.203586 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:16.203614 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:16.206019 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:16.209096 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:16.209103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:16.209109 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:16.209612 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:16.209990 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:16.209995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:16.209998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:16.210144 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:16.210880 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:16.210885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:16.211037 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 23:27:16.211230 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 23:27:16.211728 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":169}
I20260620 23:27:16.213739 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.197179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.0.97.61:36039)
0620 23:27:16.197367 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.197375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.197413 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:27:16.197619 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.197624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.197703 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.197808 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.199457 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.200456 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.202140 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.202850 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.202969 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.208040 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:16.208045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:16.208048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:16.208079 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:16.211004 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.211665 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.211678 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.211681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.211834 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.212267 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.212277 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.212283 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.212655 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:16.212904 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:16.213200 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 23:27:16.213430 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:27:16.218339 23406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8: Bootstrap starting.
I20260620 23:27:16.218485 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:27:16.222136 23406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8: Neither blocks nor log segments found. Creating new log.
I20260620 23:27:16.223273 23406 log.cc:824] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8: Log is configured to *not* fsync() on all Append() calls
I20260620 23:27:16.221463 23233 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2eed25a5b096434fba55e066b9c33ead"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35118
I20260620 23:27:16.226080 23406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8: No bootstrap required, opened a new log
I20260620 23:27:16.233325 23406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } }
I20260620 23:27:16.233719 23406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170171543817485dbf7e54b53d6cfec8, State: Initialized, Role: FOLLOWER
I20260620 23:27:16.234167 23406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } }
I20260620 23:27:16.235221 23412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } } }
I20260620 23:27:16.235625 23412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:16.236362 23406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:27:16.245956 23423 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:27:16.246090 23423 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:27:16.244095 23339 master_runner.cc:186] Error getting master registration for 127.0.97.62:38773: OK, instance_id { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" instance_seqno: 1781998035117696 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:27:16.255594 23339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:27:16.260558 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } attempt: 4
Jun 20 23:27:16 dist-test-slave-5kzz krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:27:16.278365 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.261357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35408)
0620 23:27:16.261679 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:16.261700 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:16.261840 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:16.262267 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:16.262271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:16.262853 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:16.263078 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:16.263092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.264615 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.264623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:16.266184 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:16.266203 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.267290 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.267300 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:16.267523 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.268259 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:16.268284 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:16.271367 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:16.275728 (+ 4361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:16.275733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:16.275738 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:16.276150 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:16.276559 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:16.276564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:16.276568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:16.276732 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:16.277288 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:16.277292 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:16.277469 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 23:27:16.277612 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 23:27:16.277960 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260620 23:27:16.278354 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.261431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.0.97.60:44115)
0620 23:27:16.261654 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.261661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.261781 (+ 120us) server_negotiation.cc:407] Connection header received
0620 23:27:16.261999 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.262004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.262097 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.262199 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.263241 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.264413 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.266365 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.267123 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.267244 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.271624 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:16.271630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:16.271635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:16.271686 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:16.275548 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.276279 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.276285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.276289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.276427 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.276848 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.276854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.276857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.277158 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:16.277332 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:16.277732 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 23:27:16.277957 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:27:16.285151 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead: Bootstrap starting.
I20260620 23:27:16.289666 23258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead: Neither blocks nor log segments found. Creating new log.
I20260620 23:27:16.291241 23258 log.cc:824] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead: Log is configured to *not* fsync() on all Append() calls
I20260620 23:27:16.295321 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead: No bootstrap required, opened a new log
I20260620 23:27:16.303577 23258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } }
I20260620 23:27:16.304069 23258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eed25a5b096434fba55e066b9c33ead, State: Initialized, Role: FOLLOWER
I20260620 23:27:16.304646 23258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } }
I20260620 23:27:16.305728 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } } }
I20260620 23:27:16.306114 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:16.307152 23258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:27:16.307236 23233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:27:16.307437 23233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:27:16.310179 23233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } }
I20260620 23:27:16.311861 23233 leader_election.cc:288] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [CANDIDATE]: Term 1 election: Requested vote from peers 76708ab1fa454894a3a810f3b742340a (127.0.97.61:36039), 170171543817485dbf7e54b53d6cfec8 (127.0.97.60:44115)
I20260620 23:27:16.313311 23381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2eed25a5b096434fba55e066b9c33ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "170171543817485dbf7e54b53d6cfec8"
I20260620 23:27:16.313863 23381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:27:16.318656 23381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eed25a5b096434fba55e066b9c33ead in term 1.
I20260620 23:27:16.319608 23209 leader_election.cc:302] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170171543817485dbf7e54b53d6cfec8, 2eed25a5b096434fba55e066b9c33ead; no voters:
I20260620 23:27:16.320372 23426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:27:16.324330 23426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [term 1 LEADER]: Becoming Leader. State: Replica: 2eed25a5b096434fba55e066b9c33ead, State: Running, Role: LEADER
I20260620 23:27:16.325574 23426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } }
I20260620 23:27:16.338084 23433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eed25a5b096434fba55e066b9c33ead. Latest consensus state: current_term: 1 leader_uuid: "2eed25a5b096434fba55e066b9c33ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } } }
I20260620 23:27:16.338938 23433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:16.348538 23439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260620 23:27:16.355691 23212 tablet.cc:2369] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:27:16.366053 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } attempt: 5
W20260620 23:27:16.378055 23360 tablet.cc:2369] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:27:16.383009 23439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:27:16.395143 23449 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Jun 20 23:27:16 dist-test-slave-5kzz krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:27:16.398990 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.357072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.0.97.62:38773)
0620 23:27:16.357261 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.357267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.358963 (+ 1696us) server_negotiation.cc:407] Connection header received
0620 23:27:16.359443 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.359450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.359549 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.359632 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.364675 (+ 5043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.365858 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.370793 (+ 4935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.371580 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.371703 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.379684 (+ 7981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:16.379693 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:16.379698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:16.379754 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:16.383836 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.394749 (+ 10913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.394758 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.394763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.395035 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.396053 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.396059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.396062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.396441 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:16.396715 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:16.397626 (+ 911us) server_negotiation.cc:299] Negotiation successful
0620 23:27:16.398343 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:27:16.411631 23381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [term 1 FOLLOWER]: Refusing update from remote peer 2eed25a5b096434fba55e066b9c33ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:27:16.414790 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.356621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.97.60:44115)
0620 23:27:16.356876 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.356885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.366822 (+ 9937us) server_negotiation.cc:407] Connection header received
0620 23:27:16.367343 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.367352 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.367439 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.367524 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.370908 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.372472 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.383844 (+ 11372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.384892 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.385018 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.402570 (+ 17552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:16.402579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:16.402585 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:16.402630 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:16.406579 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.410164 (+ 3585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.410170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.410174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.410344 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.410885 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.410891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.410895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.411332 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:16.411584 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:16.414001 (+ 2417us) server_negotiation.cc:299] Negotiation successful
0620 23:27:16.414338 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 23:27:16.400185 23191 master_runner.cc:186] Error getting master registration for 127.0.97.61:36039: OK, instance_id { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" instance_seqno: 1781998035634183 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:27:16.416491 23191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:27:16.421213 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.366740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44115 (local address 127.0.0.1:35432)
0620 23:27:16.367139 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 23:27:16.367156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:27:16.367269 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:27:16.386789 (+ 19520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:27:16.386796 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:27:16.387312 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:27:16.387545 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:16.387559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.389531 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.389536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:16.391001 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:27:16.391018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.392416 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.392423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:27:16.392641 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.393250 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:27:16.393275 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:27:16.396653 (+ 3378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:27:16.416690 (+ 20037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:16.416697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:16.416701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:16.417305 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:16.417901 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:27:16.417906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:27:16.417910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:27:16.418120 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:27:16.418809 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:27:16.418815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:27:16.418981 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 23:27:16.420321 (+ 1340us) client_negotiation.cc:240] Negotiation successful
0620 23:27:16.420670 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20260620 23:27:16.422844 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.367460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.0.97.60:44115)
0620 23:27:16.386251 (+ 18791us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.386257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.386282 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:27:16.386344 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.386351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.386438 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.387716 (+ 1278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.388037 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.389330 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.391241 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.391946 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.392966 (+ 1020us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.396900 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:16.396908 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:16.396913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:16.396957 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:16.415796 (+ 18839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.417528 (+ 1732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.417533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.417538 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.417719 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.418275 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.418280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.418284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.418626 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:16.421948 (+ 3322us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:16.422111 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:27:16.422361 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18634,"thread_start_us":19067,"threads_started":1}
I20260620 23:27:16.422468 23433 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:27:16.428891 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:16.358520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.0.97.61:36039)
0620 23:27:16.358759 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.358768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.390459 (+ 31691us) server_negotiation.cc:407] Connection header received
0620 23:27:16.398063 (+ 7604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.398068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.398160 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.398240 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.402748 (+ 4508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.403906 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.412105 (+ 8199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:16.412843 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:16.412963 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:16.417676 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:27:16.417683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:27:16.417688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:27:16.417732 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:27:16.423060 (+ 5328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.424693 (+ 1633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.424700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.424704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.425051 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:27:16.425873 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:27:16.425879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:27:16.425907 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:27:16.426528 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:27:16.426943 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:16.427930 (+ 987us) server_negotiation.cc:299] Negotiation successful
0620 23:27:16.428358 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:27:16.429703 23332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a: Bootstrap starting.
I20260620 23:27:16.438774 23332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a: Neither blocks nor log segments found. Creating new log.
I20260620 23:27:16.440341 23332 log.cc:824] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:27:16.445490 23412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eed25a5b096434fba55e066b9c33ead. Latest consensus state: current_term: 1 leader_uuid: "2eed25a5b096434fba55e066b9c33ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } } }
I20260620 23:27:16.445889 23412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:16.450240 23451 mvcc.cc:205] Tried to move back new op lower bound from 7299063957129420800 to 7299063956802539520. Current Snapshot: MvccSnapshot[applied={T|T < 7299063957129420800}]
I20260620 23:27:16.450894 23332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a: No bootstrap required, opened a new log
I20260620 23:27:16.456034 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2eed25a5b096434fba55e066b9c33ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } } }
I20260620 23:27:16.456399 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:16.457638 23439 catalog_manager.cc:1347] Generated new cluster ID: 4cf6d1b20abe44439532558354e36b2a
I20260620 23:27:16.457741 23439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:27:16.464170 23412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2eed25a5b096434fba55e066b9c33ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } } }
I20260620 23:27:16.464473 23412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:16.465583 23332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } }
I20260620 23:27:16.466106 23332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76708ab1fa454894a3a810f3b742340a, State: Initialized, Role: FOLLOWER
I20260620 23:27:16.466739 23332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } }
I20260620 23:27:16.474936 23456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } } }
I20260620 23:27:16.477411 23456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:16.477331 23307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2eed25a5b096434fba55e066b9c33ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "76708ab1fa454894a3a810f3b742340a"
I20260620 23:27:16.477850 23307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:27:16.477267 23332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:27:16.481628 23307 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eed25a5b096434fba55e066b9c33ead in term 1.
I20260620 23:27:16.482646 23306 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [term 1 FOLLOWER]: Refusing update from remote peer 2eed25a5b096434fba55e066b9c33ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:27:16.485230 23433 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:27:16.507975 23439 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:27:16.509833 23439 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:27:16.534406 23456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eed25a5b096434fba55e066b9c33ead. Latest consensus state: current_term: 1 leader_uuid: "2eed25a5b096434fba55e066b9c33ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } } }
I20260620 23:27:16.534785 23456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:16.534591 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2eed25a5b096434fba55e066b9c33ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } } }
I20260620 23:27:16.534905 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead [sys.catalog]: This master's current role is: LEADER
I20260620 23:27:16.535696 23439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2eed25a5b096434fba55e066b9c33ead: Generated new TSK 0
I20260620 23:27:16.535501 23464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2eed25a5b096434fba55e066b9c33ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eed25a5b096434fba55e066b9c33ead" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "76708ab1fa454894a3a810f3b742340a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36039 } } peers { permanent_uuid: "170171543817485dbf7e54b53d6cfec8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44115 } } }
I20260620 23:27:16.535991 23464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:27:16.536830 23439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:27:16.591423 23473 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 23:27:16.641383 23260 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:16.628226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.0.97.62:38773)
0620 23:27:16.628702 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.628711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.638798 (+ 10087us) server_negotiation.cc:407] Connection header received
0620 23:27:16.639676 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.639682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.639776 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.639843 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.641119 (+ 1276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35140: BlockingRecv error: recv got EOF from 127.0.0.1:35140 (error 108)
Metrics: {"server-negotiator.queue_time_us":304}
W20260620 23:27:16.641659 23448 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:16.628131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.0.97.60:44115)
0620 23:27:16.628485 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.628495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.638875 (+ 10380us) server_negotiation.cc:407] Connection header received
0620 23:27:16.639532 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.639542 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.639682 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.639760 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.641453 (+ 1693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35448: BlockingRecv error: recv got EOF from 127.0.0.1:35448 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
W20260620 23:27:16.643764 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:16.635062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.0.97.61:36039)
0620 23:27:16.635313 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.635322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.640679 (+ 5357us) server_negotiation.cc:407] Connection header received
0620 23:27:16.642358 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.642366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.642469 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.642560 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.643535 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60646: BlockingRecv error: recv got EOF from 127.0.0.1:60646 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 23:27:16.676918 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:16.672624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.0.97.61:36039)
0620 23:27:16.672807 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.672814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.675282 (+ 2468us) server_negotiation.cc:407] Connection header received
0620 23:27:16.675521 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.675527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.675682 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.675803 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.676646 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60658: BlockingRecv error: recv got EOF from 127.0.0.1:60658 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 23:27:16.688934 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:16.685262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.0.97.61:36039)
0620 23:27:16.685443 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.685450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.687278 (+ 1828us) server_negotiation.cc:407] Connection header received
0620 23:27:16.687498 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.687505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.687608 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.687726 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.688710 (+ 984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60672: BlockingRecv error: recv got EOF from 127.0.0.1:60672 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 23:27:16.697048 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:16.694760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.0.97.61:36039)
0620 23:27:16.694944 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.694950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.695464 (+ 514us) server_negotiation.cc:407] Connection header received
0620 23:27:16.695678 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.695684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.695792 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.695907 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.696782 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60684: BlockingRecv error: recv got EOF from 127.0.0.1:60684 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 23:27:16.706712 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:16.704491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.0.97.61:36039)
0620 23:27:16.704724 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.704731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.705251 (+ 520us) server_negotiation.cc:407] Connection header received
0620 23:27:16.705442 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.705448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.705546 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.705654 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.706462 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60700: BlockingRecv error: recv got EOF from 127.0.0.1:60700 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260620 23:27:16.721726 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:16.719435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.0.97.61:36039)
0620 23:27:16.719675 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.719681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.720195 (+ 514us) server_negotiation.cc:407] Connection header received
0620 23:27:16.720391 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.720397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.720498 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.720613 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.721475 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60706: BlockingRecv error: recv got EOF from 127.0.0.1:60706 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 23:27:16.744038 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:16.741379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.0.97.61:36039)
0620 23:27:16.741602 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.741611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.742472 (+ 861us) server_negotiation.cc:407] Connection header received
0620 23:27:16.742712 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.742719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.742815 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.742928 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.743792 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60708: BlockingRecv error: recv got EOF from 127.0.0.1:60708 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 23:27:16.781755 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:16.779401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.0.97.61:36039)
0620 23:27:16.779800 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.779807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.780289 (+ 482us) server_negotiation.cc:407] Connection header received
0620 23:27:16.780480 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.780486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.780583 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.780697 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.781488 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60712: BlockingRecv error: recv got EOF from 127.0.0.1:60712 (error 108)
Metrics: {"server-negotiator.queue_time_us":278}
W20260620 23:27:16.851533 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:16.849478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.0.97.61:36039)
0620 23:27:16.849698 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.849705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.850086 (+ 381us) server_negotiation.cc:407] Connection header received
0620 23:27:16.850308 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.850317 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.850391 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.850513 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.851286 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60720: BlockingRecv error: recv got EOF from 127.0.0.1:60720 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 23:27:16.985531 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:16.983672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.0.97.61:36039)
0620 23:27:16.983845 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:16.983850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:16.984109 (+ 259us) server_negotiation.cc:407] Connection header received
0620 23:27:16.984282 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:16.984286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:16.984366 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:16.984519 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:16.985324 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60734: BlockingRecv error: recv got EOF from 127.0.0.1:60734 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 23:27:17.246876 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:27:17.245056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.0.97.61:36039)
0620 23:27:17.245279 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:17.245286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:17.245462 (+ 176us) server_negotiation.cc:407] Connection header received
0620 23:27:17.245691 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:17.245695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:17.245765 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:17.245897 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:27:17.246625 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60738: BlockingRecv error: recv got EOF from 127.0.0.1:60738 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:27:17.248363 23423 catalog_manager.cc:1259] Loaded cluster ID: 4cf6d1b20abe44439532558354e36b2a
I20260620 23:27:17.248476 23423 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8: loading cluster ID for follower catalog manager: success
I20260620 23:27:17.250921 23423 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8: acquiring CA information for follower catalog manager: success
I20260620 23:27:17.252591 23423 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 170171543817485dbf7e54b53d6cfec8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:27:17.595649 23473 catalog_manager.cc:1259] Loaded cluster ID: 4cf6d1b20abe44439532558354e36b2a
I20260620 23:27:17.595835 23473 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a: loading cluster ID for follower catalog manager: success
I20260620 23:27:17.598387 23473 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a: acquiring CA information for follower catalog manager: success
I20260620 23:27:17.600250 23473 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 76708ab1fa454894a3a810f3b742340a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:27:17.770481 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:17.763334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60744 (local address 127.0.97.61:36039)
0620 23:27:17.764008 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:17.764015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:17.764274 (+ 259us) server_negotiation.cc:407] Connection header received
0620 23:27:17.764521 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:17.764527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:17.764680 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:17.764799 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:17.765306 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:17.766183 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:17.767998 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:17.768518 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:17.768633 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:17.769227 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:17.769764 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:17.769867 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:17.770030 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:27:17.770103 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":330,"threads_started":1}
I20260620 23:27:17.786906 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:17.780448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.0.97.60:44115)
0620 23:27:17.781101 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:17.781106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:17.781129 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:27:17.781252 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:17.781255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:17.781428 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:17.781646 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:17.782048 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:17.782830 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:17.784337 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:17.784835 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:17.784999 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:17.785527 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:17.786252 (+ 725us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:17.786428 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:17.786482 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 23:27:17.786566 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":427,"threads_started":1}
I20260620 23:27:17.816562 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:17.800288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.0.97.60:44115)
0620 23:27:17.800790 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:17.800799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:17.808624 (+ 7825us) server_negotiation.cc:407] Connection header received
0620 23:27:17.808845 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:17.808851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:17.808967 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:17.809110 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:17.810290 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:17.811525 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:17.814073 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:17.814803 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:17.814926 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:17.815639 (+ 713us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:17.815798 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:17.815867 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:17.816119 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 23:27:17.816221 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260620 23:27:17.819274 23551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:17.799846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.0.97.62:38773)
0620 23:27:17.807729 (+ 7883us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:17.807738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:17.808841 (+ 1103us) server_negotiation.cc:407] Connection header received
0620 23:27:17.808996 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:17.809001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:17.809262 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:17.809485 (+ 223us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:17.810606 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:17.812077 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:17.815248 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:17.816196 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:17.816356 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:17.817209 (+ 853us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:17.817809 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:17.817946 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:17.818631 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 23:27:17.818780 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7626,"thread_start_us":7513,"threads_started":1}
I20260620 23:27:17.825337 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:27:17.800158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.0.97.61:36039)
0620 23:27:17.800597 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 23:27:17.800608 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:27:17.818838 (+ 18230us) server_negotiation.cc:407] Connection header received
0620 23:27:17.819143 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:27:17.819149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:27:17.819238 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:27:17.819299 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:27:17.819864 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:17.821078 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:17.823370 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:27:17.823881 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:27:17.824051 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:27:17.824473 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:27:17.824582 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:27:17.824658 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:27:17.824870 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 23:27:17.824967 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260620 23:27:17.830439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 23191
I20260620 23:27:17.831506 23253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:27:17.943562 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 23191
W20260620 23:27:17.969942 23355 connection.cc:535] server connection from 127.0.0.1:35408 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:27:17.970431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 23265
I20260620 23:27:17.971659 23327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:27:18.084057 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 23265
I20260620 23:27:18.104373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskaeh70X/build/asan/bin/kudu pid 23339
I20260620 23:27:18.105537 23401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:27:18.214412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaeh70X/build/asan/bin/kudu with pid 23339
2026-06-20T23:27:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4953 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9227 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (357638 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST